Oct 30 15:35:16.409130 ip-10-0-133-222 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 15:35:16.409141 ip-10-0-133-222 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 15:35:16.409150 ip-10-0-133-222 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 15:35:16.409486 ip-10-0-133-222 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 15:35:26.592584 ip-10-0-133-222 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 15:35:26.592599 ip-10-0-133-222 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7026c9de7ad54f12a74988febb768551 -- Oct 30 15:37:29.857623 ip-10-0-133-222 systemd[1]: Starting Kubernetes Kubelet... Oct 30 15:37:30.249679 ip-10-0-133-222 kubenswrapper[2607]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 15:37:30.249679 ip-10-0-133-222 kubenswrapper[2607]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 15:37:30.249679 ip-10-0-133-222 kubenswrapper[2607]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 15:37:30.249679 ip-10-0-133-222 kubenswrapper[2607]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257449 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257532 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257536 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257539 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257542 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257545 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257548 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257550 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.257541 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257554 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257557 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257560 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257563 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257566 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257569 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257571 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257574 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257577 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257580 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257583 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257587 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257592 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257595 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257599 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257604 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257606 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257609 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257612 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.257828 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257615 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257618 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257621 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257624 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257626 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257632 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257635 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257638 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257641 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257644 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257647 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257649 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257652 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257655 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257657 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257661 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257664 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257666 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257669 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.258275 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257672 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257674 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257677 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257680 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257684 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257687 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257691 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257694 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257696 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257699 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257702 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257704 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257707 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257710 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257712 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257715 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257718 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257720 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257723 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257726 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.258736 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257729 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257731 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257734 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257737 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257739 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257742 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257745 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257747 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257750 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257752 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257755 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257757 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257760 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257763 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257765 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257768 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257770 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257773 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257776 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257778 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.259215 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257781 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257784 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257786 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257789 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257791 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257794 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257796 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257799 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257801 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257804 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257807 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257810 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257813 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257816 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257820 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257823 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257826 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257828 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257831 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.257834 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.259708 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259509 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259518 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259521 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259525 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259529 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259532 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259535 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259538 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259542 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259546 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259554 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259557 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259560 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259563 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259567 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259571 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259574 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259578 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259581 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.260179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259584 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259586 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259589 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259592 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259594 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259597 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259600 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259602 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259605 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259608 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259610 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259613 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259616 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259618 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259621 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259624 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259626 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259629 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259632 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.260696 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259634 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259637 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259640 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259643 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259645 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259649 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259651 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259655 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259657 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259660 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259662 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259665 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259668 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259670 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259673 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259676 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259678 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259681 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259684 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.261147 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259686 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259689 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259691 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259694 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259697 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259700 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259702 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259705 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259708 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259711 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259713 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259716 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259719 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259722 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259725 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259727 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259731 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259733 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259736 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259739 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.261615 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259742 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259744 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259747 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259749 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259752 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259755 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259758 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259760 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259763 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259772 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259776 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259778 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259781 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259783 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259786 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259788 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259791 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259793 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259796 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259799 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.262075 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259802 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259805 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259807 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259810 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259813 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259816 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259820 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259823 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.259825 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260404 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260411 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260415 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260419 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260423 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260426 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260429 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260432 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260435 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260437 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260440 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.262558 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260442 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260445 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260448 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260451 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260453 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260456 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260458 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260461 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260463 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260466 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260469 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260472 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260475 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260478 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260480 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260483 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260486 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260489 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260491 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260494 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.263027 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260496 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260499 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260501 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260504 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260507 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260510 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260512 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260515 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260517 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260520 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260522 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260525 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260528 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260531 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260533 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260536 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260538 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260541 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260543 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260546 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.263512 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260548 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260551 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260554 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260558 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260561 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260564 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260567 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260570 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260572 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260575 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260578 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260580 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260583 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260585 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260588 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260590 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260593 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260596 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260598 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.263983 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260601 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260605 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260608 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260611 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260613 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260616 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260619 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260622 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260624 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260627 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260630 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260632 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260635 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260637 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260640 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260645 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260647 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260650 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260652 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.264532 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260655 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260657 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260660 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260663 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260666 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260668 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260670 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260673 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260676 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260678 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260680 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260683 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260686 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260688 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260691 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260694 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.260696 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260792 2607 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260799 2607 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260805 2607 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260810 2607 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 15:37:30.264987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260814 2607 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260817 2607 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260824 2607 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260829 2607 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260832 2607 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260835 2607 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260847 2607 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260856 2607 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260859 2607 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260862 2607 flags.go:64] FLAG: --cgroup-root="" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260865 2607 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260868 2607 flags.go:64] FLAG: --client-ca-file="" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260871 2607 flags.go:64] FLAG: --cloud-provider="external" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260874 2607 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260880 2607 flags.go:64] FLAG: --cluster-domain="" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260883 2607 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260886 2607 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260889 2607 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260893 2607 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260897 2607 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260900 2607 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260903 2607 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260906 2607 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260909 2607 flags.go:64] FLAG: --contention-profiling="false" Oct 30 15:37:30.265529 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260913 2607 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260916 2607 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260919 2607 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260922 2607 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260926 2607 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260929 2607 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260932 2607 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260936 2607 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260939 2607 flags.go:64] FLAG: --enable-server="true" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260942 2607 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260946 2607 flags.go:64] FLAG: --event-burst="100" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260949 2607 flags.go:64] FLAG: --event-qps="50" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260952 2607 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260955 2607 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260958 2607 flags.go:64] FLAG: --eviction-hard="" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260964 2607 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260967 2607 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260970 2607 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260973 2607 flags.go:64] FLAG: --eviction-soft="" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260976 2607 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260979 2607 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260982 2607 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260985 2607 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260988 2607 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260991 2607 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 15:37:30.266107 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260994 2607 flags.go:64] FLAG: --feature-gates="" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.260998 2607 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261002 2607 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261005 2607 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261008 2607 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261011 2607 flags.go:64] FLAG: --healthz-port="10248" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261014 2607 flags.go:64] FLAG: --help="false" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261017 2607 flags.go:64] FLAG: --hostname-override="ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261021 2607 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261024 2607 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261027 2607 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261030 2607 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261033 2607 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261036 2607 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261040 2607 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261043 2607 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261046 2607 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261049 2607 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261052 2607 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261054 2607 flags.go:64] FLAG: --kube-reserved="" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261058 2607 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261060 2607 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261065 2607 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261068 2607 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 15:37:30.266732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261071 2607 flags.go:64] FLAG: --lock-file="" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261074 2607 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261077 2607 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261080 2607 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261086 2607 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261089 2607 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261092 2607 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261095 2607 flags.go:64] FLAG: --logging-format="text" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261102 2607 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261106 2607 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261109 2607 flags.go:64] FLAG: --manifest-url="" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261112 2607 flags.go:64] FLAG: --manifest-url-header="" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261116 2607 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261119 2607 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261123 2607 flags.go:64] FLAG: --max-pods="110" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261126 2607 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261129 2607 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261133 2607 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261136 2607 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261139 2607 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261142 2607 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261145 2607 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261156 2607 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261159 2607 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261162 2607 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 15:37:30.267310 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261165 2607 flags.go:64] FLAG: --pod-cidr="" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261168 2607 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261171 2607 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261175 2607 flags.go:64] FLAG: --pods-per-core="0" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261178 2607 flags.go:64] FLAG: --port="10250" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261182 2607 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261185 2607 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0949fc1c082626899" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261188 2607 flags.go:64] FLAG: --qos-reserved="" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261191 2607 flags.go:64] FLAG: --read-only-port="10255" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261195 2607 flags.go:64] FLAG: --register-node="true" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261197 2607 flags.go:64] FLAG: --register-with-taints="" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261201 2607 flags.go:64] FLAG: --registry-burst="10" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261204 2607 flags.go:64] FLAG: --registry-qps="5" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261207 2607 flags.go:64] FLAG: --reserved-cpus="" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261210 2607 flags.go:64] FLAG: --reserved-memory="" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261213 2607 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261216 2607 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261219 2607 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261222 2607 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261225 2607 flags.go:64] FLAG: --runonce="false" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261228 2607 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261231 2607 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261234 2607 flags.go:64] FLAG: --seccomp-default="false" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261237 2607 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261240 2607 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261243 2607 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 15:37:30.267917 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261246 2607 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261249 2607 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261252 2607 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261257 2607 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261259 2607 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261262 2607 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261266 2607 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261269 2607 flags.go:64] FLAG: --system-cgroups="" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261272 2607 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261276 2607 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261279 2607 flags.go:64] FLAG: --tls-cert-file="" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261284 2607 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261288 2607 flags.go:64] FLAG: --tls-min-version="" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261291 2607 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261294 2607 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261297 2607 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261299 2607 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261302 2607 flags.go:64] FLAG: --v="2" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261307 2607 flags.go:64] FLAG: --version="false" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261311 2607 flags.go:64] FLAG: --vmodule="" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261315 2607 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261319 2607 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261469 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261474 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.268546 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261477 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261480 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261482 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261485 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261488 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261490 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261493 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261496 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261499 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261502 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261505 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261509 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261512 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261515 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261518 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261520 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261523 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261526 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261528 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261532 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.269152 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261535 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261538 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261540 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261543 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261546 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261548 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261551 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261553 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261556 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261560 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261563 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261566 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261569 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261572 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261574 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261577 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261579 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261582 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261584 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.269646 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261587 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261590 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261593 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261595 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261599 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261601 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261604 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261606 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261609 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261612 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261614 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261617 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261621 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261624 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261627 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261630 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261632 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261635 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261637 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261640 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.270102 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261643 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261645 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261648 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261650 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261653 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261655 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261658 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261661 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261664 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261666 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261669 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261671 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261674 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261676 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261679 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261682 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261688 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261692 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261696 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.270603 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261699 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261701 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261704 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261707 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261709 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261713 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261716 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261719 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261721 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261724 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261726 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261729 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261732 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261734 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261737 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261740 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261742 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261745 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261747 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261750 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.271052 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261752 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261755 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261757 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261760 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261762 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261765 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.261790 2607 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261829 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261833 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261838 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261841 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261844 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261846 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261849 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261851 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261854 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.271540 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261857 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261860 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261866 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261868 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261871 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261874 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261876 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261879 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261881 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261884 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261887 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261889 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261892 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261894 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261897 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261899 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261902 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261905 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261907 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261910 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.271936 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261912 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261915 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261919 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261922 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261926 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261930 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261934 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261936 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261939 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261942 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261944 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261947 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261950 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261952 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261956 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261959 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261961 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261964 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261967 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261969 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.272418 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261972 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261974 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261977 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261979 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261982 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261985 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261987 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261990 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261993 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261995 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.261998 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262000 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262003 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262005 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262008 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262011 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262013 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262017 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262020 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.272888 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262023 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262025 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262028 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262030 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262033 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262035 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262038 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262042 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262045 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262049 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262053 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262056 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262059 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262061 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262064 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262067 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262069 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262072 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262074 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.273336 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262077 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262080 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262082 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262085 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262088 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262090 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262093 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262096 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262098 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262100 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262103 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262106 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262109 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262111 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262114 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262117 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262119 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262122 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.262125 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.270560 2607 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 15:37:30.273804 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.270575 2607 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.270590 2607 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.270595 2607 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270653 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270658 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270662 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270665 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270668 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270671 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270675 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270679 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270682 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270685 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270688 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270690 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270693 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270696 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270698 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270702 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.274304 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270704 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270707 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270710 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270712 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270715 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270718 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270721 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270725 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270727 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270730 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270733 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270736 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270739 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270742 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270745 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270748 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270751 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270753 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270756 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270758 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.274755 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270761 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270764 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270766 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270769 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270772 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270774 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270777 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270780 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270782 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270785 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270787 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270790 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270792 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270795 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270797 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270800 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270803 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270805 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270808 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270811 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.275224 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270814 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270818 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270820 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270823 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270826 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270829 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270831 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270834 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270837 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270839 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270842 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270844 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270847 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270850 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270852 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270855 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270859 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270862 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270866 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.275807 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270869 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270872 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270874 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270877 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270880 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270882 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270885 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270887 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270890 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270892 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270895 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270897 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270900 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270902 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270906 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270909 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270911 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270914 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270917 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.276255 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270920 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270922 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270925 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270927 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270930 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270933 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270935 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270938 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270940 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270943 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270945 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.270948 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.270974 2607 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271112 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271117 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271119 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.276701 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271122 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271125 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271127 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271130 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271132 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271135 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271137 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271140 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271142 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271145 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271148 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271150 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271154 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271156 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271159 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271162 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271164 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271167 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271170 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271172 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.277065 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271175 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271178 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271180 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271183 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271185 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271188 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271190 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271193 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271196 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271198 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271201 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271203 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271206 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271209 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271211 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271214 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271216 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271219 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271221 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.277543 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271224 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271226 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271229 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271231 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271234 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271237 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271240 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271243 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271245 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271248 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271250 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271253 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271256 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271258 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271261 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271264 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271268 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271271 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271274 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.277991 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271276 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271279 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271281 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271284 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271286 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271289 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271291 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271294 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271296 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271299 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271301 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271304 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271306 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271309 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271312 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271314 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271317 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271319 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271322 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271325 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.278447 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271328 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271330 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271333 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271335 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271338 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271340 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271343 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271363 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271367 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271370 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271372 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271375 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271377 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271380 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271383 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271386 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271388 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271391 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271393 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.278918 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271396 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271400 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271403 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271407 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271410 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271413 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.271437 2607 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271470 2607 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271474 2607 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271477 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271480 2607 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271482 2607 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271485 2607 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271487 2607 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271490 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271493 2607 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:37:30.279362 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271496 2607 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271498 2607 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271501 2607 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271503 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271506 2607 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271509 2607 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271511 2607 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271514 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271517 2607 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271519 2607 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271522 2607 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271525 2607 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271527 2607 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271530 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271532 2607 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271535 2607 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271537 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271540 2607 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271542 2607 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:37:30.279740 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271545 2607 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271550 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271553 2607 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271560 2607 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271562 2607 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271565 2607 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271568 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271571 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271573 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271576 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271578 2607 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271582 2607 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271587 2607 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271590 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271593 2607 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271596 2607 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271599 2607 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271601 2607 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271604 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271606 2607 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:37:30.280178 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271609 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271611 2607 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271614 2607 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271616 2607 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271619 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271622 2607 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271624 2607 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271627 2607 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271629 2607 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271632 2607 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271634 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271637 2607 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271639 2607 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271642 2607 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271645 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271647 2607 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271650 2607 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271652 2607 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271655 2607 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271657 2607 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:37:30.280671 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271660 2607 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271662 2607 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271665 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271668 2607 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271670 2607 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271673 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271676 2607 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271678 2607 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271680 2607 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271683 2607 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271686 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271688 2607 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271691 2607 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271693 2607 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271696 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271698 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271701 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271703 2607 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271706 2607 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:37:30.281144 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271708 2607 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271711 2607 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271713 2607 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271716 2607 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271718 2607 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271721 2607 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271724 2607 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271726 2607 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271729 2607 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271732 2607 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271734 2607 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271737 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271739 2607 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271742 2607 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271746 2607 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271749 2607 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271752 2607 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271757 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:30.271760 2607 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:37:30.281679 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.271817 2607 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 15:37:30.282158 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.273919 2607 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 15:37:30.282158 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.276397 2607 server.go:1014] "Starting client certificate rotation" Oct 30 15:37:30.282158 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.276493 2607 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 15:37:30.282158 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.276524 2607 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 15:37:30.282158 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.278620 2607 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 15:37:30.285416 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.285379 2607 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 15:37:30.298028 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.298013 2607 log.go:25] "Validated CRI v1 runtime API" Oct 30 15:37:30.300441 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.300427 2607 log.go:25] "Validated CRI v1 image API" Oct 30 15:37:30.301288 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.301254 2607 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 15:37:30.304385 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.304361 2607 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f4778c6f-330e-4507-96c0-0d63a0ac521d:/dev/nvme0n1p3 fec848c1-aa96-410d-bbed-8477a8f7db2c:/dev/nvme0n1p4] Oct 30 15:37:30.304457 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.304383 2607 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 15:37:30.310104 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.309994 2607 manager.go:217] Machine: {Timestamp:2026-10-30 15:37:30.308757347 +0000 UTC m=+0.337459587 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100346 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec212f5a4e80a5980ab14200e45be55b SystemUUID:ec212f5a-4e80-a598-0ab1-4200e45be55b BootID:7026c9de-7ad5-4f12-a749-88febb768551 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:55:13:13 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:55:13:13 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:c7:88:21:38:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 15:37:30.310104 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.310104 2607 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 15:37:30.310217 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.310179 2607 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 15:37:30.313893 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.313867 2607 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 15:37:30.314040 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.313895 2607 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 15:37:30.314081 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.314043 2607 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 15:37:30.314081 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.314051 2607 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 15:37:30.314081 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.314066 2607 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 15:37:30.314845 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.314835 2607 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 15:37:30.315618 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.315609 2607 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 15:37:30.318648 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.318636 2607 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 15:37:30.318758 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.318749 2607 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 15:37:30.319510 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.319500 2607 kubelet.go:493] "Attempting to sync node with API server" Oct 30 15:37:30.319563 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.319513 2607 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 15:37:30.319563 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.319526 2607 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 15:37:30.319563 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.319536 2607 kubelet.go:405] "Adding apiserver pod source" Oct 30 15:37:30.319563 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.319547 2607 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 15:37:30.319864 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.319849 2607 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 15:37:30.322140 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.322123 2607 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 15:37:30.327140 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.327128 2607 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 15:37:30.330617 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330604 2607 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 15:37:30.330662 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330635 2607 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 15:37:30.330784 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330775 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 15:37:30.330818 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330790 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 15:37:30.330818 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330797 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 15:37:30.330818 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330803 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 15:37:30.330818 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330809 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 15:37:30.330818 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.330815 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 15:37:30.332296 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.332286 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 15:37:30.332328 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.332297 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 15:37:30.332328 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.332304 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 15:37:30.332328 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.332311 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 15:37:30.332328 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.332320 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 15:37:30.332328 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.332328 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 15:37:30.333249 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.333240 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 15:37:30.333249 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.333249 2607 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 15:37:30.336878 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.336865 2607 server.go:1263] "Started kubelet" Oct 30 15:37:30.336981 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.336951 2607 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 15:37:30.337080 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.337038 2607 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 15:37:30.337110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.337099 2607 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 15:37:30.338246 ip-10-0-133-222 systemd[1]: Started Kubernetes Kubelet. Oct 30 15:37:30.342036 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.342000 2607 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 15:37:30.344123 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.344093 2607 server.go:317] "Adding debug handlers to kubelet server" Oct 30 15:37:30.344363 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.344327 2607 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 15:37:30.347600 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.347579 2607 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 15:37:30.348054 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.348032 2607 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 15:37:30.348360 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.348328 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:30.348743 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.348725 2607 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 15:37:30.348743 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.348745 2607 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 15:37:30.348906 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.348804 2607 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 15:37:30.348906 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.348880 2607 reconstruct.go:97] "Volume reconstruction finished" Oct 30 15:37:30.348906 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.348887 2607 reconciler.go:29] "Reconciler: start to sync state" Oct 30 15:37:30.349092 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.349076 2607 factory.go:55] Registering systemd factory Oct 30 15:37:30.349153 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.349094 2607 factory.go:223] Registration of the systemd container factory successfully Oct 30 15:37:30.349322 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.349310 2607 factory.go:153] Registering CRI-O factory Oct 30 15:37:30.349392 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.349323 2607 factory.go:223] Registration of the crio container factory successfully Oct 30 15:37:30.349392 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.349390 2607 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 15:37:30.349496 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.349409 2607 factory.go:103] Registering Raw factory Oct 30 15:37:30.349496 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.349424 2607 manager.go:1199] Started watching for new ooms in manager Oct 30 15:37:30.350440 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.350426 2607 manager.go:319] Starting recovery of all containers Oct 30 15:37:30.360397 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.360382 2607 manager.go:324] Recovery completed Oct 30 15:37:30.365548 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.365536 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:30.367196 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.367173 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:30.367283 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.367208 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:30.367283 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.367217 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:30.367784 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.367754 2607 cpu_manager.go:226] "Starting" policy="none" Oct 30 15:37:30.367784 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.367773 2607 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 15:37:30.367881 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.367792 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:30.367881 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.367794 2607 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 15:37:30.368642 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.368625 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:30.368726 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.368657 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:30.368726 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.368677 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:30.369117 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.369105 2607 policy_none.go:50] "Start" Oct 30 15:37:30.369117 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.369119 2607 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 15:37:30.369206 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.369128 2607 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 15:37:30.370137 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.370126 2607 policy_none.go:44] "Start" Oct 30 15:37:30.370991 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.370972 2607 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-222.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 15:37:30.372819 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.370964 2607 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-222.ec2.internal.18e358be26848664 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-222.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-222.ec2.internal,},FirstTimestamp:2026-10-30 15:37:30.336839268 +0000 UTC m=+0.365541508,LastTimestamp:2026-10-30 15:37:30.336839268 +0000 UTC m=+0.365541508,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-222.ec2.internal,}" Oct 30 15:37:30.379971 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.379929 2607 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 15:37:30.411316 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.411274 2607 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 15:37:30.413823 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.413809 2607 manager.go:341] "Starting Device Plugin manager" Oct 30 15:37:30.413899 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.413862 2607 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 15:37:30.413899 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.413874 2607 server.go:87] "Starting device plugin registration server" Oct 30 15:37:30.414070 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.414058 2607 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 15:37:30.414113 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.414071 2607 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 15:37:30.414210 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.414185 2607 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 15:37:30.414342 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.414301 2607 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 15:37:30.414342 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.414311 2607 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 15:37:30.414803 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.414778 2607 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 15:37:30.414866 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.414837 2607 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:30.415239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.415224 2607 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 15:37:30.423088 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.422868 2607 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-222.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 15:37:30.423088 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.422827 2607 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-222.ec2.internal.18e358be2853b9d6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-222.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-222.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-222.ec2.internal,},FirstTimestamp:2026-10-30 15:37:30.367195606 +0000 UTC m=+0.395897844,LastTimestamp:2026-10-30 15:37:30.367195606 +0000 UTC m=+0.395897844,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-222.ec2.internal,}" Oct 30 15:37:30.423088 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.422880 2607 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 15:37:30.423274 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.423169 2607 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 15:37:30.440635 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.439864 2607 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-222.ec2.internal.18e358be2853fdbf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-222.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-133-222.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-133-222.ec2.internal,},FirstTimestamp:2026-10-30 15:37:30.367212991 +0000 UTC m=+0.395915229,LastTimestamp:2026-10-30 15:37:30.367212991 +0000 UTC m=+0.395915229,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-222.ec2.internal,}" Oct 30 15:37:30.464456 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.464430 2607 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 15:37:30.465558 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.465538 2607 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 15:37:30.465558 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.465560 2607 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 15:37:30.465689 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.465576 2607 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 15:37:30.465689 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.465606 2607 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 15:37:30.465939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.465889 2607 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4xv7j" Oct 30 15:37:30.470096 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.470008 2607 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-222.ec2.internal.18e358be28541e50 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-222.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-222.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-222.ec2.internal,},FirstTimestamp:2026-10-30 15:37:30.367221328 +0000 UTC m=+0.395923568,LastTimestamp:2026-10-30 15:37:30.367221328 +0000 UTC m=+0.395923568,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-222.ec2.internal,}" Oct 30 15:37:30.481496 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.481479 2607 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4xv7j" Oct 30 15:37:30.502296 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.502248 2607 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:37:30.514189 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.514170 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:30.514918 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.514903 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:30.514987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.514930 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:30.514987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.514939 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:30.514987 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.514967 2607 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.535732 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.535590 2607 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.535826 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.535737 2607 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-222.ec2.internal\": node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:30.565771 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.565748 2607 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal"] Oct 30 15:37:30.565869 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.565805 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:30.566632 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.566617 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:30.566693 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.566649 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:30.566693 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.566659 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:30.568068 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568055 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:30.568220 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568205 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.568279 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568241 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:30.568793 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568777 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:30.568857 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568785 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:30.568857 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568824 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:30.568857 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568833 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:30.568857 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568804 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:30.568994 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.568871 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:30.569951 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.569936 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.570029 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.569959 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:30.570619 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.570604 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:30.570619 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.570628 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:30.570752 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.570636 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:30.586947 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.586923 2607 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-222.ec2.internal\" not found" node="ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.587507 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.587491 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:30.590045 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.590029 2607 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-222.ec2.internal\" not found" node="ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.651125 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.651096 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.651125 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.651123 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.651319 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.651141 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bab30e2cbbbd657a1d2e615d7ce89c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"e2bab30e2cbbbd657a1d2e615d7ce89c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.688445 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.688417 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:30.751386 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.751339 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bab30e2cbbbd657a1d2e615d7ce89c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"e2bab30e2cbbbd657a1d2e615d7ce89c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.751386 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.751386 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.751558 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.751406 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.751558 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.751452 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.751558 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.751459 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bab30e2cbbbd657a1d2e615d7ce89c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"e2bab30e2cbbbd657a1d2e615d7ce89c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.751558 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.751456 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.789057 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.789005 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:30.889296 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.889274 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:30.890414 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.890399 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.894367 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:30.894018 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:37:30.990120 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:30.990085 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:31.090586 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:31.090519 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:32.569459 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:32.569436 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:32.653057 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.653013 2607 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 15:37:32.653615 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.653186 2607 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 15:37:32.670329 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:32.670303 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:32.758846 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:32.758798 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9895a994f5b5cfb1039cd968cd927e03.slice/crio-46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5 WatchSource:0}: Error finding container 46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5: Status 404 returned error can't find the container with id 46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5 Oct 30 15:37:32.759182 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:32.759165 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bab30e2cbbbd657a1d2e615d7ce89c.slice/crio-34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510 WatchSource:0}: Error finding container 34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510: Status 404 returned error can't find the container with id 34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510 Oct 30 15:37:32.766541 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.766526 2607 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:37:32.770648 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:32.770632 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:32.847547 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.847500 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"9895a994f5b5cfb1039cd968cd927e03","Type":"ContainerStarted","Data":"46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5"} Oct 30 15:37:32.848387 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.848363 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" event={"ID":"e2bab30e2cbbbd657a1d2e615d7ce89c","Type":"ContainerStarted","Data":"34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510"} Oct 30 15:37:32.864170 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.864142 2607 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 15:32:31 +0000 UTC" deadline="2028-08-05 04:04:27.710738262 +0000 UTC" Oct 30 15:37:32.864222 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.864169 2607 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15468h26m54.846573363s" Oct 30 15:37:32.871291 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:32.871275 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:32.874420 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.874410 2607 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:37:32.875520 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.875503 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:37:32.875599 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.875532 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:37:32.875599 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:32.875544 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:37:32.971653 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:32.971625 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:33.072170 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.072102 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:33.142935 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.142903 2607 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:37:33.172841 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.172799 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:33.273624 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.273544 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:33.374143 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.374107 2607 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:37:33.430397 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.430365 2607 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:37:33.528084 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.528050 2607 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:37:33.553306 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.553280 2607 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 15:37:33.554340 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.554314 2607 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:37:33.575720 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.575692 2607 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 15:37:33.672179 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.672099 2607 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:37:33.699398 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.699375 2607 apiserver.go:52] "Watching apiserver" Oct 30 15:37:33.727238 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.727214 2607 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 15:37:33.733798 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.733765 2607 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 15:37:33.734176 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.734157 2607 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-cm4sh","openshift-multus/multus-additional-cni-plugins-974j2","kube-system/global-pull-secret-syncer-8r6df","openshift-cluster-node-tuning-operator/tuned-v8qs2","openshift-image-registry/node-ca-c99x2","openshift-multus/network-metrics-daemon-j24jl","openshift-ovn-kubernetes/ovnkube-node-2vkxh","kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal","openshift-network-diagnostics/network-check-target-tcrzp","openshift-multus/multus-wws5p","openshift-network-operator/iptables-alerter-ztxfj","kube-system/konnectivity-agent-scbpb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb"] Oct 30 15:37:33.735976 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.735785 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.737097 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.737064 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.738361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.738343 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.739908 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.739682 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.740996 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.740976 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.742462 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.742444 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.743613 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.743594 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:33.745445 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.745429 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.746937 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.746677 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:33.746937 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.746750 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:33.748549 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.748050 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.749473 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.749453 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:33.749557 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.749530 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:33.751043 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.751024 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.751137 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.751097 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:33.776358 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.776251 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 15:37:33.776779 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.776761 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 15:37:33.776779 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.776772 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 15:37:33.776946 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.776882 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 15:37:33.776946 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.776929 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 15:37:33.777056 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.776965 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 15:37:33.777056 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777049 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 15:37:33.777155 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777059 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 15:37:33.777155 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777073 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fmlw7\"" Oct 30 15:37:33.777155 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777099 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 15:37:33.777155 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777119 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 15:37:33.777155 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777144 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 15:37:33.777155 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777149 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 15:37:33.777155 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777154 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777186 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777198 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777201 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777228 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777235 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777248 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777268 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 15:37:33.777442 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777351 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 15:37:33.777794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777553 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 15:37:33.777794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777585 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 15:37:33.777794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.777665 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xbwc7\"" Oct 30 15:37:33.780109 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.780076 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 15:37:33.790361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790343 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ftvkm\"" Oct 30 15:37:33.790441 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790366 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 15:37:33.790441 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790389 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-khtmm\"" Oct 30 15:37:33.790441 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790408 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5dknm\"" Oct 30 15:37:33.790441 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790427 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 15:37:33.790658 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790541 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-84s7k\"" Oct 30 15:37:33.790658 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790614 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 15:37:33.790747 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790659 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mw526\"" Oct 30 15:37:33.790747 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790670 2607 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 15:37:33.790903 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790803 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8vjmf\"" Oct 30 15:37:33.790968 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.790918 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ck89s\"" Oct 30 15:37:33.803927 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.803906 2607 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mg955" Oct 30 15:37:33.814744 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.814722 2607 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mg955" Oct 30 15:37:33.828955 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.828933 2607 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 15:37:33.839165 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839137 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76t8z\" (UniqueName: \"kubernetes.io/projected/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-kube-api-access-76t8z\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:33.839269 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839177 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-var-lib-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.839269 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839205 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64acc66-316d-49f8-8659-a609dbcc6f63-ovn-node-metrics-cert\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.839269 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839255 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-binary-copy\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.839422 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839306 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-multus\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.839422 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839341 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-lib-modules\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.839422 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839384 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-var-lib-kubelet\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.839545 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839445 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g9dm\" (UniqueName: \"kubernetes.io/projected/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-kube-api-access-2g9dm\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.839545 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839493 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-systemd-units\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.839632 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839540 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.839632 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839570 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-socket-dir-parent\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.839632 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839617 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysconfig\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.839765 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839651 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.839765 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839677 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-netns\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.839765 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839713 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-device-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.839765 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839741 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.839978 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839779 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.839978 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839848 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.839978 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839878 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:33.839978 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839892 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-cni-binary-copy\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.839978 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839914 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-tmp\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.839978 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839966 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjsx\" (UniqueName: \"kubernetes.io/projected/7f01c529-80bc-4609-a6dd-912a16dfbba8-kube-api-access-xkjsx\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.839995 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840028 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-cnibin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840051 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-netns\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840073 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840122 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-config\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840156 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67s8z\" (UniqueName: \"kubernetes.io/projected/a64acc66-316d-49f8-8659-a609dbcc6f63-kube-api-access-67s8z\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840180 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840206 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-sys\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.840239 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840232 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-iptables-alerter-script\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840259 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-log-socket\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840286 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840345 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-system-cni-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840380 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-bin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840400 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-systemd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840429 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-ovn\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840450 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-netd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840473 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840500 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-conf-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840527 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-etc-kubernetes\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.840638 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840589 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5qwr\" (UniqueName: \"kubernetes.io/projected/da6d7847-fb28-415a-a5fb-765e87929cd4-kube-api-access-c5qwr\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840644 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-tuned\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840746 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-system-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840803 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-k8s-cni-cncf-io\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840865 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-conf\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840906 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0addea81-c333-4eed-8afa-3762350172bc-serviceca\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840935 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-tmp-dir\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.840972 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-node-log\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841004 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-multus-certs\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841029 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-hosts-file\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.841110 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841090 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841126 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-bin\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841146 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-script-lib\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841178 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-konnectivity-ca\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841214 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-kubernetes\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841255 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffz48\" (UniqueName: \"kubernetes.io/projected/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-kube-api-access-ffz48\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841319 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841359 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-kubelet-config\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841382 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-hostroot\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841405 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8wg4\" (UniqueName: \"kubernetes.io/projected/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-kube-api-access-z8wg4\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841438 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.841470 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841466 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0addea81-c333-4eed-8afa-3762350172bc-host\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841490 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-env-overrides\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841527 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841554 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-kubelet\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841594 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-slash\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841669 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-agent-certs\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841704 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-dbus\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841731 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-run\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841774 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-daemon-config\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841801 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-modprobe-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841851 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-systemd\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841899 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-host\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.841939 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841931 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cglh8\" (UniqueName: \"kubernetes.io/projected/0addea81-c333-4eed-8afa-3762350172bc-kube-api-access-cglh8\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.842430 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841965 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-host-slash\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.842430 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.841988 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-etc-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.842430 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.842012 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.842430 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.842046 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-cnibin\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.842430 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.842083 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5k9k\" (UniqueName: \"kubernetes.io/projected/f427009a-c18a-404f-9948-9453d93d132b-kube-api-access-g5k9k\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.842430 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.842109 2607 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-kubelet\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.943408 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943334 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-config\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.943408 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943371 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67s8z\" (UniqueName: \"kubernetes.io/projected/a64acc66-316d-49f8-8659-a609dbcc6f63-kube-api-access-67s8z\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.943408 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943399 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943421 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-sys\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943443 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-iptables-alerter-script\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943463 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-log-socket\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943485 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943506 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-system-cni-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943528 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-bin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943547 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-systemd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943567 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-ovn\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943590 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-netd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943613 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.943660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943644 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-conf-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943666 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-etc-kubernetes\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943689 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5qwr\" (UniqueName: \"kubernetes.io/projected/da6d7847-fb28-415a-a5fb-765e87929cd4-kube-api-access-c5qwr\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943715 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-tuned\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943742 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-system-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943766 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-k8s-cni-cncf-io\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943789 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-conf\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943813 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0addea81-c333-4eed-8afa-3762350172bc-serviceca\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943854 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-tmp-dir\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943937 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-ovn\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.943990 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-node-log\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944019 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-multus-certs\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944048 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-hosts-file\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944072 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944097 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-bin\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944122 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-script-lib\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944147 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-konnectivity-ca\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944173 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-kubernetes\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.944190 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944191 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-tmp-dir\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944197 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffz48\" (UniqueName: \"kubernetes.io/projected/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-kube-api-access-ffz48\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944227 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944256 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-kubelet-config\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944264 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-netd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944262 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944280 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-hostroot\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944306 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-conf-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944311 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wg4\" (UniqueName: \"kubernetes.io/projected/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-kube-api-access-z8wg4\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944324 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-hostroot\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944361 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944390 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0addea81-c333-4eed-8afa-3762350172bc-host\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944415 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-env-overrides\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944439 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944465 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-kubelet\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944498 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-slash\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944523 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-agent-certs\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944547 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-dbus\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.944989 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944571 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-run\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944596 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-daemon-config\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944621 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-modprobe-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944646 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-systemd\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944669 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-host\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944694 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cglh8\" (UniqueName: \"kubernetes.io/projected/0addea81-c333-4eed-8afa-3762350172bc-kube-api-access-cglh8\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944719 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-host-slash\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944760 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-etc-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944786 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944814 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-cnibin\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944811 2607 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944845 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-k8s-cni-cncf-io\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944860 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5k9k\" (UniqueName: \"kubernetes.io/projected/f427009a-c18a-404f-9948-9453d93d132b-kube-api-access-g5k9k\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944894 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-kubelet\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944909 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-config\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944914 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76t8z\" (UniqueName: \"kubernetes.io/projected/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-kube-api-access-76t8z\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944974 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-var-lib-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.945794 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.944995 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64acc66-316d-49f8-8659-a609dbcc6f63-ovn-node-metrics-cert\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945016 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-binary-copy\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945032 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-multus\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945049 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-lib-modules\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945053 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-system-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945063 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-var-lib-kubelet\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945080 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-kubelet\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945086 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9dm\" (UniqueName: \"kubernetes.io/projected/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-kube-api-access-2g9dm\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945106 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-systemd-units\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945122 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945137 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-socket-dir-parent\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945150 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-etc-kubernetes\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945159 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysconfig\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945167 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-conf\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945181 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945214 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-netns\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945243 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-device-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.946660 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945259 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945267 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945305 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945335 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945355 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945367 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945397 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-cni-binary-copy\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945400 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-kubernetes\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945422 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-tmp\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945446 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjsx\" (UniqueName: \"kubernetes.io/projected/7f01c529-80bc-4609-a6dd-912a16dfbba8-kube-api-access-xkjsx\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945471 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945493 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-cnibin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945521 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-netns\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945552 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945598 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945615 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-konnectivity-ca\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945652 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.947465 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945652 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-kubelet-config\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945683 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-var-lib-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945707 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0addea81-c333-4eed-8afa-3762350172bc-host\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.945965 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946015 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-sys\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946124 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-env-overrides\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.946230 2607 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.946321 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret podName:5a9c1136-39de-45c4-ba36-d8a4b524f252 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:34.446302274 +0000 UTC m=+3.096529574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret") pod "global-pull-secret-syncer-8r6df" (UID: "5a9c1136-39de-45c4-ba36-d8a4b524f252") : object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946492 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-kubelet\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946503 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-iptables-alerter-script\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946544 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-slash\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946559 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-log-socket\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946609 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946707 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-dbus\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946752 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-run\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946789 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-binary-copy\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946871 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-multus\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.948260 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.946976 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-lib-modules\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947033 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-var-lib-kubelet\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947269 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-daemon-config\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947285 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-systemd-units\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947346 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947415 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-socket-dir-parent\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947439 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-modprobe-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947466 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysconfig\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947515 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-systemd\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947555 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947574 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-host\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947606 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-netns\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947654 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-device-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947915 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-multus-certs\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.947981 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-system-cni-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948024 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-bin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948067 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-systemd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948093 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-node-log\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.949073 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948110 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-cnibin\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948149 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-etc-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948153 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948400 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-host-slash\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948453 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-netns\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948502 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-cnibin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948583 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0addea81-c333-4eed-8afa-3762350172bc-serviceca\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948658 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948788 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-hosts-file\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.948893 2607 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.948954 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs podName:46e6b33b-39c7-4f7d-a3ba-4ce7c4337171 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:34.448938079 +0000 UTC m=+3.099165358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs") pod "network-metrics-daemon-j24jl" (UID: "46e6b33b-39c7-4f7d-a3ba-4ce7c4337171") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.948999 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-bin\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.949554 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.949604 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-cni-binary-copy\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.949808 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.949767 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-tuned\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.950419 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.949884 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64acc66-316d-49f8-8659-a609dbcc6f63-ovn-node-metrics-cert\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.950419 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.949984 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-agent-certs\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:33.950419 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.950397 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-script-lib\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.950779 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.950731 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.950887 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.950873 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-tmp\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.972718 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.972643 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5qwr\" (UniqueName: \"kubernetes.io/projected/da6d7847-fb28-415a-a5fb-765e87929cd4-kube-api-access-c5qwr\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:37:33.972718 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.972643 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5k9k\" (UniqueName: \"kubernetes.io/projected/f427009a-c18a-404f-9948-9453d93d132b-kube-api-access-g5k9k\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:33.973417 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.973393 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76t8z\" (UniqueName: \"kubernetes.io/projected/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-kube-api-access-76t8z\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:33.975093 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.975070 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wg4\" (UniqueName: \"kubernetes.io/projected/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-kube-api-access-z8wg4\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:33.978191 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.978169 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 15:37:33.978291 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.978195 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 15:37:33.978291 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.978210 2607 projected.go:196] Error preparing data for projected volume kube-api-access-86d7k for pod openshift-network-diagnostics/network-check-target-tcrzp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:33.978393 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:33.978338 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k podName:4581006f-4626-475a-9942-da3b81d5d93b nodeName:}" failed. No retries permitted until 2026-10-30 15:37:34.478317713 +0000 UTC m=+3.128544989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-86d7k" (UniqueName: "kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k") pod "network-check-target-tcrzp" (UID: "4581006f-4626-475a-9942-da3b81d5d93b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:33.980915 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.980889 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67s8z\" (UniqueName: \"kubernetes.io/projected/a64acc66-316d-49f8-8659-a609dbcc6f63-kube-api-access-67s8z\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:33.983665 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.983641 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9dm\" (UniqueName: \"kubernetes.io/projected/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-kube-api-access-2g9dm\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:33.986920 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.986893 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffz48\" (UniqueName: \"kubernetes.io/projected/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-kube-api-access-ffz48\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:33.987023 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.987004 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjsx\" (UniqueName: \"kubernetes.io/projected/7f01c529-80bc-4609-a6dd-912a16dfbba8-kube-api-access-xkjsx\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:33.988327 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:33.988295 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cglh8\" (UniqueName: \"kubernetes.io/projected/0addea81-c333-4eed-8afa-3762350172bc-kube-api-access-cglh8\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:34.055645 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.055607 2607 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:37:34.060000 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.059795 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:34.060872 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.060852 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wws5p" Oct 30 15:37:34.072268 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.072029 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6d7847_fb28_415a_a5fb_765e87929cd4.slice/crio-7dc7d8257e7a660bd02a1bf20250db5c4a13152bcc6c3c7b46954361d49b7dae WatchSource:0}: Error finding container 7dc7d8257e7a660bd02a1bf20250db5c4a13152bcc6c3c7b46954361d49b7dae: Status 404 returned error can't find the container with id 7dc7d8257e7a660bd02a1bf20250db5c4a13152bcc6c3c7b46954361d49b7dae Oct 30 15:37:34.072681 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.072664 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:37:34.073633 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.073616 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:37:34.080972 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.080950 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:37:34.083419 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.083393 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f01c529_80bc_4609_a6dd_912a16dfbba8.slice/crio-28ad7854edbf25c99b0b385639f98172aa0d1023541e4912959581c22ab73454 WatchSource:0}: Error finding container 28ad7854edbf25c99b0b385639f98172aa0d1023541e4912959581c22ab73454: Status 404 returned error can't find the container with id 28ad7854edbf25c99b0b385639f98172aa0d1023541e4912959581c22ab73454 Oct 30 15:37:34.088041 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.088017 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6651f4_00bb_4e63_94c3_0e7884f24b4a.slice/crio-cf84e736f8eee113721132641b91d614a6bef25e3eae3723bd10b13c17ab3e10 WatchSource:0}: Error finding container cf84e736f8eee113721132641b91d614a6bef25e3eae3723bd10b13c17ab3e10: Status 404 returned error can't find the container with id cf84e736f8eee113721132641b91d614a6bef25e3eae3723bd10b13c17ab3e10 Oct 30 15:37:34.090047 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.089902 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:37:34.094248 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.094215 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0addea81_c333_4eed_8afa_3762350172bc.slice/crio-e743ea90bf277a5dd9ebcaafa6f9fd870a95126799d362b9c586de73e0668c31 WatchSource:0}: Error finding container e743ea90bf277a5dd9ebcaafa6f9fd870a95126799d362b9c586de73e0668c31: Status 404 returned error can't find the container with id e743ea90bf277a5dd9ebcaafa6f9fd870a95126799d362b9c586de73e0668c31 Oct 30 15:37:34.098360 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.098323 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod395236fd_641c_4bd3_8f77_3e0ac6cd29c0.slice/crio-19e38ef81b3fd35a26dca2df79bd7e7b723860f6254d983baaad45afdcea253e WatchSource:0}: Error finding container 19e38ef81b3fd35a26dca2df79bd7e7b723860f6254d983baaad45afdcea253e: Status 404 returned error can't find the container with id 19e38ef81b3fd35a26dca2df79bd7e7b723860f6254d983baaad45afdcea253e Oct 30 15:37:34.098448 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.098439 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:34.106297 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.106103 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3e40fa_3cc7_4e68_b204_98ff6f6e9d07.slice/crio-40743ccae829831cc26adc1b80a6ba2ff49ee6c1991c3b786cf3724e54ba8647 WatchSource:0}: Error finding container 40743ccae829831cc26adc1b80a6ba2ff49ee6c1991c3b786cf3724e54ba8647: Status 404 returned error can't find the container with id 40743ccae829831cc26adc1b80a6ba2ff49ee6c1991c3b786cf3724e54ba8647 Oct 30 15:37:34.113576 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.113558 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:37:34.114629 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.114611 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:37:34.121880 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.121859 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf427009a_c18a_404f_9948_9453d93d132b.slice/crio-b82a9fb94dcfe4e5a0cc1292dd1fed24a03311f1d4839eefbefe96cf2681c332 WatchSource:0}: Error finding container b82a9fb94dcfe4e5a0cc1292dd1fed24a03311f1d4839eefbefe96cf2681c332: Status 404 returned error can't find the container with id b82a9fb94dcfe4e5a0cc1292dd1fed24a03311f1d4839eefbefe96cf2681c332 Oct 30 15:37:34.126837 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:37:34.126796 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d9f9bb_0f4f_4110_9ad8_28ac0bbde949.slice/crio-577e4ca0aa157429408ed80b8d5ccec4cd9c261ed056554926b52346cbd7d77b WatchSource:0}: Error finding container 577e4ca0aa157429408ed80b8d5ccec4cd9c261ed056554926b52346cbd7d77b: Status 404 returned error can't find the container with id 577e4ca0aa157429408ed80b8d5ccec4cd9c261ed056554926b52346cbd7d77b Oct 30 15:37:34.450332 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.449527 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:34.450332 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.449802 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:34.450332 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.449749 2607 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:34.450332 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.450002 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs podName:46e6b33b-39c7-4f7d-a3ba-4ce7c4337171 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:35.449985036 +0000 UTC m=+4.100212314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs") pod "network-metrics-daemon-j24jl" (UID: "46e6b33b-39c7-4f7d-a3ba-4ce7c4337171") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:34.450332 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.449943 2607 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:34.450332 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.450199 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret podName:5a9c1136-39de-45c4-ba36-d8a4b524f252 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:35.45017968 +0000 UTC m=+4.100406961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret") pod "global-pull-secret-syncer-8r6df" (UID: "5a9c1136-39de-45c4-ba36-d8a4b524f252") : object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:34.551135 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.551094 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:34.551297 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.551262 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 15:37:34.551297 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.551280 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 15:37:34.551297 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.551291 2607 projected.go:196] Error preparing data for projected volume kube-api-access-86d7k for pod openshift-network-diagnostics/network-check-target-tcrzp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:34.551454 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.551346 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k podName:4581006f-4626-475a-9942-da3b81d5d93b nodeName:}" failed. No retries permitted until 2026-10-30 15:37:35.551328071 +0000 UTC m=+4.201555353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-86d7k" (UniqueName: "kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k") pod "network-check-target-tcrzp" (UID: "4581006f-4626-475a-9942-da3b81d5d93b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:34.816977 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.816813 2607 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 15:32:33 +0000 UTC" deadline="2028-05-08 05:01:06.806311079 +0000 UTC" Oct 30 15:37:34.816977 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.816881 2607 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13333h23m31.989434467s" Oct 30 15:37:34.844754 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.844267 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:34.844754 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:34.844392 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:34.867531 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.866691 2607 generic.go:356] "Generic (PLEG): container finished" podID="9895a994f5b5cfb1039cd968cd927e03" containerID="0fc07cac45980a200ac4cbf0d51434c473c1c95d4d307c2683493aee8f335407" exitCode=0 Oct 30 15:37:34.867531 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.866775 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"9895a994f5b5cfb1039cd968cd927e03","Type":"ContainerDied","Data":"0fc07cac45980a200ac4cbf0d51434c473c1c95d4d307c2683493aee8f335407"} Oct 30 15:37:34.881264 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.881210 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cm4sh" event={"ID":"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949","Type":"ContainerStarted","Data":"577e4ca0aa157429408ed80b8d5ccec4cd9c261ed056554926b52346cbd7d77b"} Oct 30 15:37:34.884184 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.883686 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerStarted","Data":"b82a9fb94dcfe4e5a0cc1292dd1fed24a03311f1d4839eefbefe96cf2681c332"} Oct 30 15:37:34.887369 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.887312 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-scbpb" event={"ID":"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07","Type":"ContainerStarted","Data":"40743ccae829831cc26adc1b80a6ba2ff49ee6c1991c3b786cf3724e54ba8647"} Oct 30 15:37:34.907635 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.907608 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" event={"ID":"0b6651f4-00bb-4e63-94c3-0e7884f24b4a","Type":"ContainerStarted","Data":"cf84e736f8eee113721132641b91d614a6bef25e3eae3723bd10b13c17ab3e10"} Oct 30 15:37:34.922156 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.922129 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wws5p" event={"ID":"da6d7847-fb28-415a-a5fb-765e87929cd4","Type":"ContainerStarted","Data":"7dc7d8257e7a660bd02a1bf20250db5c4a13152bcc6c3c7b46954361d49b7dae"} Oct 30 15:37:34.934445 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.934416 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"6dc9934ae87fb3d89a43ae47192a7a3ecc0d172429002481617440605b8d2d45"} Oct 30 15:37:34.958534 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.958503 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" event={"ID":"e2bab30e2cbbbd657a1d2e615d7ce89c","Type":"ContainerStarted","Data":"821d54b7e84961c16737a40ac2642c5b566da446cc753a8311ca1121b505f3e2"} Oct 30 15:37:34.968137 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.968083 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ztxfj" event={"ID":"395236fd-641c-4bd3-8f77-3e0ac6cd29c0","Type":"ContainerStarted","Data":"19e38ef81b3fd35a26dca2df79bd7e7b723860f6254d983baaad45afdcea253e"} Oct 30 15:37:34.987267 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.987238 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c99x2" event={"ID":"0addea81-c333-4eed-8afa-3762350172bc","Type":"ContainerStarted","Data":"e743ea90bf277a5dd9ebcaafa6f9fd870a95126799d362b9c586de73e0668c31"} Oct 30 15:37:34.997002 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:34.996945 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" event={"ID":"7f01c529-80bc-4609-a6dd-912a16dfbba8","Type":"ContainerStarted","Data":"28ad7854edbf25c99b0b385639f98172aa0d1023541e4912959581c22ab73454"} Oct 30 15:37:35.461452 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:35.461418 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:35.461648 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:35.461517 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:35.461705 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.461655 2607 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:35.461740 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.461714 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs podName:46e6b33b-39c7-4f7d-a3ba-4ce7c4337171 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:37.461696504 +0000 UTC m=+6.111923786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs") pod "network-metrics-daemon-j24jl" (UID: "46e6b33b-39c7-4f7d-a3ba-4ce7c4337171") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:35.462053 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.461972 2607 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:35.462053 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.462031 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret podName:5a9c1136-39de-45c4-ba36-d8a4b524f252 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:37.462015993 +0000 UTC m=+6.112243269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret") pod "global-pull-secret-syncer-8r6df" (UID: "5a9c1136-39de-45c4-ba36-d8a4b524f252") : object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:35.562872 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:35.562840 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:35.563046 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.563000 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 15:37:35.563046 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.563024 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 15:37:35.563046 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.563036 2607 projected.go:196] Error preparing data for projected volume kube-api-access-86d7k for pod openshift-network-diagnostics/network-check-target-tcrzp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:35.563203 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.563090 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k podName:4581006f-4626-475a-9942-da3b81d5d93b nodeName:}" failed. No retries permitted until 2026-10-30 15:37:37.563073424 +0000 UTC m=+6.213300702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-86d7k" (UniqueName: "kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k") pod "network-check-target-tcrzp" (UID: "4581006f-4626-475a-9942-da3b81d5d93b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:35.817725 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:35.817561 2607 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 15:32:33 +0000 UTC" deadline="2028-07-22 09:47:04.492246136 +0000 UTC" Oct 30 15:37:35.817725 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:35.817599 2607 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15138h9m28.674652118s" Oct 30 15:37:35.847361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:35.846570 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:35.847361 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.846703 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:35.847361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:35.847107 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:35.847361 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:35.847218 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:36.034935 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:36.034894 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"9895a994f5b5cfb1039cd968cd927e03","Type":"ContainerStarted","Data":"ed82353a60a77c0259a7a20bbf808af36d9ed16a414acb05978c7096b9c928f1"} Oct 30 15:37:36.078071 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:36.077974 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" podStartSLOduration=3.077954718 podStartE2EDuration="3.077954718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:35.000235436 +0000 UTC m=+3.650462737" watchObservedRunningTime="2026-10-30 15:37:36.077954718 +0000 UTC m=+4.728182003" Oct 30 15:37:36.845012 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:36.844978 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:36.845549 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:36.845113 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:37.044433 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.044394 2607 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="e226b1847904a4d9dbbc3b5bb64ed306e2a710d9c47b7ba6d6ffb3bb27d1a2f2" exitCode=0 Oct 30 15:37:37.047202 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.047087 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerDied","Data":"e226b1847904a4d9dbbc3b5bb64ed306e2a710d9c47b7ba6d6ffb3bb27d1a2f2"} Oct 30 15:37:37.124796 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.124731 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" podStartSLOduration=4.124713876 podStartE2EDuration="4.124713876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:36.07889376 +0000 UTC m=+4.729121046" watchObservedRunningTime="2026-10-30 15:37:37.124713876 +0000 UTC m=+5.774941161" Oct 30 15:37:37.477866 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.477771 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:37.477866 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.477900 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:37.478130 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.478042 2607 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:37.478130 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.478107 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs podName:46e6b33b-39c7-4f7d-a3ba-4ce7c4337171 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:41.478086265 +0000 UTC m=+10.128313536 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs") pod "network-metrics-daemon-j24jl" (UID: "46e6b33b-39c7-4f7d-a3ba-4ce7c4337171") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:37.478560 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.478536 2607 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:37.478663 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.478601 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret podName:5a9c1136-39de-45c4-ba36-d8a4b524f252 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:41.478585593 +0000 UTC m=+10.128812870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret") pod "global-pull-secret-syncer-8r6df" (UID: "5a9c1136-39de-45c4-ba36-d8a4b524f252") : object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:37.578842 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.578797 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:37.579052 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.579033 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 15:37:37.579138 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.579062 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 15:37:37.579138 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.579075 2607 projected.go:196] Error preparing data for projected volume kube-api-access-86d7k for pod openshift-network-diagnostics/network-check-target-tcrzp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:37.579138 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.579130 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k podName:4581006f-4626-475a-9942-da3b81d5d93b nodeName:}" failed. No retries permitted until 2026-10-30 15:37:41.579113094 +0000 UTC m=+10.229340363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-86d7k" (UniqueName: "kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k") pod "network-check-target-tcrzp" (UID: "4581006f-4626-475a-9942-da3b81d5d93b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:37.847534 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.846976 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:37.847534 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.847088 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:37.847534 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:37.847193 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:37.847534 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:37.847285 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:38.050211 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:38.050137 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-scbpb" event={"ID":"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07","Type":"ContainerStarted","Data":"abe5b2c5191deec511441c04715e14426cc7f5f89caa983ba2f02ecd535b476c"} Oct 30 15:37:38.094296 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:38.094234 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-scbpb" podStartSLOduration=2.399892392 podStartE2EDuration="6.09421456s" totalImagesPullingTime="3.694322168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:38.093968482 +0000 UTC m=+6.744195767" watchObservedRunningTime="2026-10-30 15:37:38.09421456 +0000 UTC m=+6.744441848" Oct 30 15:37:38.845020 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:38.844993 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:38.845165 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:38.845135 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:39.846875 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:39.846843 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:39.847275 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:39.846941 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:39.847275 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:39.847264 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:39.847347 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:39.847334 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:40.058898 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:40.058863 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-c99x2" event={"ID":"0addea81-c333-4eed-8afa-3762350172bc","Type":"ContainerStarted","Data":"6c837be69ffacc5dd0daaab5516bb2fdf540a7d7ca1d952631d1c259febd109c"} Oct 30 15:37:40.062086 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:40.062054 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" event={"ID":"7f01c529-80bc-4609-a6dd-912a16dfbba8","Type":"ContainerStarted","Data":"2df53eb5b5ef39add35bb8980924cab51b79ae7f7a95e5f1b7ab0c11474af6fe"} Oct 30 15:37:40.101488 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:40.101385 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-c99x2" podStartSLOduration=4.348926572 podStartE2EDuration="9.101366591s" totalImagesPullingTime="4.752440019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:40.10061774 +0000 UTC m=+8.750845025" watchObservedRunningTime="2026-10-30 15:37:40.101366591 +0000 UTC m=+8.751593876" Oct 30 15:37:40.845028 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:40.844946 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:40.845183 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:40.845129 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:41.104984 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.104942 2607 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 15:37:41.509882 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.508542 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:41.509882 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.508594 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:41.509882 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.508722 2607 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:41.509882 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.508780 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret podName:5a9c1136-39de-45c4-ba36-d8a4b524f252 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:49.50876334 +0000 UTC m=+18.158990605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret") pod "global-pull-secret-syncer-8r6df" (UID: "5a9c1136-39de-45c4-ba36-d8a4b524f252") : object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:41.509882 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.509222 2607 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:41.509882 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.509274 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs podName:46e6b33b-39c7-4f7d-a3ba-4ce7c4337171 nodeName:}" failed. No retries permitted until 2026-10-30 15:37:49.509258173 +0000 UTC m=+18.159485440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs") pod "network-metrics-daemon-j24jl" (UID: "46e6b33b-39c7-4f7d-a3ba-4ce7c4337171") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:41.609551 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.609513 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:41.609767 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.609701 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 15:37:41.609767 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.609729 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 15:37:41.609767 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.609744 2607 projected.go:196] Error preparing data for projected volume kube-api-access-86d7k for pod openshift-network-diagnostics/network-check-target-tcrzp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:41.610123 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.609805 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k podName:4581006f-4626-475a-9942-da3b81d5d93b nodeName:}" failed. No retries permitted until 2026-10-30 15:37:49.609784958 +0000 UTC m=+18.260012223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-86d7k" (UniqueName: "kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k") pod "network-check-target-tcrzp" (UID: "4581006f-4626-475a-9942-da3b81d5d93b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:41.797578 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.797450 2607 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T15:37:41.104959595Z","UUID":"17ee5233-f0e4-4430-9dcf-b0660e379c15","Handler":null,"Name":"","Endpoint":""} Oct 30 15:37:41.808347 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.799991 2607 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 15:37:41.808347 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.800020 2607 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 15:37:41.845942 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.845916 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:41.846057 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.846028 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:41.846108 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:41.846086 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:41.846206 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:41.846150 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:42.069089 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.069057 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cm4sh" event={"ID":"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949","Type":"ContainerStarted","Data":"222cc7c1fc715be58e70b4c4733fc8b79893e540f893bc1ab88958488a3d602d"} Oct 30 15:37:42.072784 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.071685 2607 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="e906f031d7cf286e6273e62e3f61c4e14259012ee1cb7935522810116ead5431" exitCode=0 Oct 30 15:37:42.072784 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.071765 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerDied","Data":"e906f031d7cf286e6273e62e3f61c4e14259012ee1cb7935522810116ead5431"} Oct 30 15:37:42.075612 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.075186 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wws5p" event={"ID":"da6d7847-fb28-415a-a5fb-765e87929cd4","Type":"ContainerStarted","Data":"bcba021e77d183555654cd1bf9c2b054ea291f7d88a3e05d8d6f4604ee760c8f"} Oct 30 15:37:42.077667 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.077599 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" event={"ID":"7f01c529-80bc-4609-a6dd-912a16dfbba8","Type":"ContainerStarted","Data":"29abcf3093d19c89c5365e5f6433b9e198f8c6bc25d1702ca528de44e23479e5"} Oct 30 15:37:42.111814 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.111757 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-cm4sh" podStartSLOduration=2.397621677 podStartE2EDuration="10.111744014s" totalImagesPullingTime="7.714122337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:42.111366188 +0000 UTC m=+10.761593472" watchObservedRunningTime="2026-10-30 15:37:42.111744014 +0000 UTC m=+10.761971298" Oct 30 15:37:42.160817 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.157886 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wws5p" podStartSLOduration=3.865698632 podStartE2EDuration="11.157845634s" totalImagesPullingTime="7.292147002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:42.156728379 +0000 UTC m=+10.806955665" watchObservedRunningTime="2026-10-30 15:37:42.157845634 +0000 UTC m=+10.808072919" Oct 30 15:37:42.711281 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.710725 2607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:42.711281 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.711270 2607 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:42.844298 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:42.844231 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:42.844451 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:42.844340 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:43.081296 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.081263 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ztxfj" event={"ID":"395236fd-641c-4bd3-8f77-3e0ac6cd29c0","Type":"ContainerStarted","Data":"844f40b54fe5a89f8ad6e695588bf8ec979b4808a7a2c4831733019fa2f609c3"} Oct 30 15:37:43.083862 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.083836 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" event={"ID":"7f01c529-80bc-4609-a6dd-912a16dfbba8","Type":"ContainerStarted","Data":"172ee0df2403a2882648e2e5803585dcdf77fa78d56bc7368d184974ca135736"} Oct 30 15:37:43.086408 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.086381 2607 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="0a42003726e627a21ee9295751c6cf53e800848c83a4750c5d3fd27ead4a295b" exitCode=0 Oct 30 15:37:43.086634 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.086606 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerDied","Data":"0a42003726e627a21ee9295751c6cf53e800848c83a4750c5d3fd27ead4a295b"} Oct 30 15:37:43.126749 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.126702 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ztxfj" podStartSLOduration=4.583873178 podStartE2EDuration="12.126689025s" totalImagesPullingTime="7.542815847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:43.126218796 +0000 UTC m=+11.776446080" watchObservedRunningTime="2026-10-30 15:37:43.126689025 +0000 UTC m=+11.776916309" Oct 30 15:37:43.222208 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.222149 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" podStartSLOduration=4.518158993 podStartE2EDuration="12.222131317s" totalImagesPullingTime="7.703972324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:43.221914548 +0000 UTC m=+11.872141832" watchObservedRunningTime="2026-10-30 15:37:43.222131317 +0000 UTC m=+11.872358602" Oct 30 15:37:43.844757 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.844726 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:43.844950 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:43.844769 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:43.844950 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:43.844882 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:43.845064 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:43.844995 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:44.844834 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:44.844785 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:44.845638 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:44.844913 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:45.094094 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:45.094059 2607 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="91559b52c3b3a62bb00b00c72ec15d29d11968fb2102fe47b99701cf34806c9c" exitCode=0 Oct 30 15:37:45.094275 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:45.094150 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerDied","Data":"91559b52c3b3a62bb00b00c72ec15d29d11968fb2102fe47b99701cf34806c9c"} Oct 30 15:37:45.096037 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:45.096003 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" event={"ID":"0b6651f4-00bb-4e63-94c3-0e7884f24b4a","Type":"ContainerStarted","Data":"66374e52c8a39463be950315ac8a880c7ec4b3e5be8109ea97a3db8e26b318e5"} Oct 30 15:37:45.216170 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:45.216115 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" podStartSLOduration=4.258626594 podStartE2EDuration="14.216097574s" totalImagesPullingTime="9.95747098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:45.215799945 +0000 UTC m=+13.866027229" watchObservedRunningTime="2026-10-30 15:37:45.216097574 +0000 UTC m=+13.866324861" Oct 30 15:37:45.844315 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:45.844281 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:45.844315 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:45.844322 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:45.844554 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:45.844424 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:45.844554 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:45.844519 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:46.844749 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:46.844528 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:46.845135 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:46.844847 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:47.109578 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:47.109542 2607 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="7a7989e4d01d471661699617f142eae3369d60cc6d66aa0d4a94fa0196b5311e" exitCode=0 Oct 30 15:37:47.109725 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:47.109596 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerDied","Data":"7a7989e4d01d471661699617f142eae3369d60cc6d66aa0d4a94fa0196b5311e"} Oct 30 15:37:47.335978 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:47.335950 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:47.336184 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:47.336167 2607 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 15:37:47.336747 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:47.336722 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:37:47.844904 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:47.844881 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:47.845348 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:47.844978 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:47.845348 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:47.845005 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:47.845348 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:47.845052 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:48.114064 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:48.114033 2607 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="b6de40bbd8e19b12c2bf59bf2e50e7a6ec10e4edcf019e8728595f8be4cc631b" exitCode=0 Oct 30 15:37:48.114212 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:48.114111 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerDied","Data":"b6de40bbd8e19b12c2bf59bf2e50e7a6ec10e4edcf019e8728595f8be4cc631b"} Oct 30 15:37:48.845257 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:48.845103 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:48.845709 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:48.845349 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:49.118405 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.118322 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"f858ae130c04a33cf5c21647f56987b346f84d4018875b34ae39a83d6436ce82"} Oct 30 15:37:49.118405 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.118396 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"9fc65e797ae5dfe15425197a149e9bf21d7dd47c2a40ebad3bf68c4a1af223b4"} Oct 30 15:37:49.118575 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.118433 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"30cfe5abd713412f16488e3b11c09f2adf9a251cdc2b31e756fdfade5822a9d7"} Oct 30 15:37:49.118575 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.118459 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"21e70b1c0b653d71eeaf3f64b68f8cfb23fc08cfa3876fb5b6dab49a48a718b4"} Oct 30 15:37:49.118575 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.118493 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"815f2636ff512a7815b2cb8dcc769a580d033ab170ac5451dedf067044d7b79f"} Oct 30 15:37:49.118575 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.118538 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"e56fe8ea3ef05b805f761e7131dec39f245435d7b591dbd4e396cb87cbebba45"} Oct 30 15:37:49.124343 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.124312 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-974j2" event={"ID":"f427009a-c18a-404f-9948-9453d93d132b","Type":"ContainerStarted","Data":"8dbe59ca514d57fa9311b70defda37ec9618ec80d00d7a235d7f981f054380ec"} Oct 30 15:37:49.173431 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.173390 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-974j2" podStartSLOduration=7.4997254810000005 podStartE2EDuration="17.173379243s" totalImagesPullingTime="9.673653762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:49.172773714 +0000 UTC m=+17.823001008" watchObservedRunningTime="2026-10-30 15:37:49.173379243 +0000 UTC m=+17.823606526" Oct 30 15:37:49.566871 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.566789 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:49.566871 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.566837 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:49.567026 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.566917 2607 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:49.567026 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.566948 2607 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:49.567026 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.566972 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs podName:46e6b33b-39c7-4f7d-a3ba-4ce7c4337171 nodeName:}" failed. No retries permitted until 2026-10-30 15:38:05.56695612 +0000 UTC m=+34.217183382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs") pod "network-metrics-daemon-j24jl" (UID: "46e6b33b-39c7-4f7d-a3ba-4ce7c4337171") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 15:37:49.567026 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.566988 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret podName:5a9c1136-39de-45c4-ba36-d8a4b524f252 nodeName:}" failed. No retries permitted until 2026-10-30 15:38:05.566981659 +0000 UTC m=+34.217208921 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret") pod "global-pull-secret-syncer-8r6df" (UID: "5a9c1136-39de-45c4-ba36-d8a4b524f252") : object "kube-system"/"original-pull-secret" not registered Oct 30 15:37:49.667302 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.667268 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:49.667453 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.667398 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 15:37:49.667453 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.667415 2607 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 15:37:49.667453 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.667425 2607 projected.go:196] Error preparing data for projected volume kube-api-access-86d7k for pod openshift-network-diagnostics/network-check-target-tcrzp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:49.667622 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.667481 2607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k podName:4581006f-4626-475a-9942-da3b81d5d93b nodeName:}" failed. No retries permitted until 2026-10-30 15:38:05.667464188 +0000 UTC m=+34.317691450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-86d7k" (UniqueName: "kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k") pod "network-check-target-tcrzp" (UID: "4581006f-4626-475a-9942-da3b81d5d93b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 15:37:49.844615 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.844540 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:49.844615 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:49.844576 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:49.844779 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.844662 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:49.844819 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:49.844779 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:50.845189 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:50.845162 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:50.845623 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:50.845253 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:51.131400 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:51.131364 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"c68e958b1955f384e50e049c6030ea4c9f1f5304d75004bdb423f48091b2e9d3"} Oct 30 15:37:51.845587 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:51.845557 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:51.846056 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:51.845647 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:51.846056 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:51.846013 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:51.849075 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:51.847154 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:52.844474 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:52.844448 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:52.844627 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:52.844548 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:53.138671 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:53.138640 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" event={"ID":"a64acc66-316d-49f8-8659-a609dbcc6f63","Type":"ContainerStarted","Data":"388d5383e722930818c7f7dee3225c4f7908e93e68ed4e8154b74147f488f0ca"} Oct 30 15:37:53.139298 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:53.138920 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:53.139298 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:53.138987 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:53.153878 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:53.153860 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:53.245144 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:53.245100 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" podStartSLOduration=8.015403561 podStartE2EDuration="22.245088393s" totalImagesPullingTime="14.229684832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:53.24487485 +0000 UTC m=+21.895102134" watchObservedRunningTime="2026-10-30 15:37:53.245088393 +0000 UTC m=+21.895315676" Oct 30 15:37:53.844460 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:53.844424 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:53.844643 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:53.844537 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:53.844643 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:53.844595 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:53.844756 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:53.844668 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:54.141410 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.141378 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:54.156149 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.156126 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:37:54.844402 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.844331 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:54.844576 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:54.844435 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:54.847399 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.847374 2607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8r6df"] Oct 30 15:37:54.850916 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.850897 2607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tcrzp"] Oct 30 15:37:54.851002 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.850981 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:54.851070 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:54.851055 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:54.851395 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.851378 2607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j24jl"] Oct 30 15:37:54.851471 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:54.851461 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:54.851548 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:54.851534 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:55.142699 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:55.142670 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:55.143211 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:55.142782 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:56.844793 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:56.844766 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:56.845216 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:56.844897 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:56.845216 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:56.844914 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:37:56.845216 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:56.844934 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:56.845216 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:56.845046 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:56.845216 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:56.845124 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:58.845328 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:58.845297 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:37:58.845328 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:58.845325 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:37:58.845751 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:58.845400 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:37:58.845751 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:37:58.845453 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:37:58.845751 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:58.845527 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:37:58.845751 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:37:58.845584 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:38:00.844602 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:00.844459 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:00.844902 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:00.844466 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:00.844902 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:38:00.844679 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8r6df" podUID="5a9c1136-39de-45c4-ba36-d8a4b524f252" Oct 30 15:38:00.844902 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:38:00.844747 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tcrzp" podUID="4581006f-4626-475a-9942-da3b81d5d93b" Oct 30 15:38:00.844902 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:00.844466 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:00.844902 ip-10-0-133-222 kubenswrapper[2607]: E1030 15:38:00.844849 2607 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j24jl" podUID="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" Oct 30 15:38:01.122400 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:01.122372 2607 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeReady" Oct 30 15:38:01.122576 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:01.122516 2607 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 15:38:02.844484 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.844448 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:02.844988 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.844579 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:02.844988 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.844614 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:02.875172 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.875150 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qtl5k\"" Oct 30 15:38:02.875297 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.875184 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5ljdx\"" Oct 30 15:38:02.875361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.875197 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 15:38:02.875361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.875227 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nlpvl\"" Oct 30 15:38:02.875361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.875242 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 15:38:02.875361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.875258 2607 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 15:38:02.875361 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:02.875260 2607 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 15:38:04.114681 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:04.114647 2607 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-scbpb_3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:04.291596 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:04.291566 2607 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-scbpb_3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:04.567170 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:04.567135 2607 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-scbpb_3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:04.757629 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:04.757555 2607 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-scbpb_3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/konnectivity-agent/0.log" limit=1024 Oct 30 15:38:05.583516 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.583485 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:05.583516 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.583521 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:05.587068 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.587049 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:05.587152 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.587098 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:05.626927 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.626906 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:05.652070 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.652049 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:05.684572 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.684548 2607 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:05.686877 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.686860 2607 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:05.966626 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:05.966604 2607 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:06.060738 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:06.060678 2607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j24jl"] Oct 30 15:38:06.064092 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:38:06.064069 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e6b33b_39c7_4f7d_a3ba_4ce7c4337171.slice/crio-2b48ee46f793ce7b9d046ff088609679e18db41d13c10946a887cb0c856a8cb1 WatchSource:0}: Error finding container 2b48ee46f793ce7b9d046ff088609679e18db41d13c10946a887cb0c856a8cb1: Status 404 returned error can't find the container with id 2b48ee46f793ce7b9d046ff088609679e18db41d13c10946a887cb0c856a8cb1 Oct 30 15:38:06.071514 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:06.071493 2607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8r6df"] Oct 30 15:38:06.077179 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:38:06.077156 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9c1136_39de_45c4_ba36_d8a4b524f252.slice/crio-45d9cc5ff6a236f9d4f67746ad303844fddb6388eafe9930f6eb134c66666947 WatchSource:0}: Error finding container 45d9cc5ff6a236f9d4f67746ad303844fddb6388eafe9930f6eb134c66666947: Status 404 returned error can't find the container with id 45d9cc5ff6a236f9d4f67746ad303844fddb6388eafe9930f6eb134c66666947 Oct 30 15:38:06.163875 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:06.163846 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j24jl" event={"ID":"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171","Type":"ContainerStarted","Data":"2b48ee46f793ce7b9d046ff088609679e18db41d13c10946a887cb0c856a8cb1"} Oct 30 15:38:06.164867 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:06.164845 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8r6df" event={"ID":"5a9c1136-39de-45c4-ba36-d8a4b524f252","Type":"ContainerStarted","Data":"45d9cc5ff6a236f9d4f67746ad303844fddb6388eafe9930f6eb134c66666947"} Oct 30 15:38:06.171406 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:06.171390 2607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cm4sh_31d9f9bb-0f4f-4110-9ad8-28ac0bbde949/dns-node-resolver/0.log" Oct 30 15:38:06.267292 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:06.267234 2607 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tcrzp"] Oct 30 15:38:06.270203 ip-10-0-133-222 kubenswrapper[2607]: W1030 15:38:06.270181 2607 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4581006f_4626_475a_9942_da3b81d5d93b.slice/crio-104f1869a31d0e686f4c19a918f0d93a8887a334d114680a5ab649b84f28fb63 WatchSource:0}: Error finding container 104f1869a31d0e686f4c19a918f0d93a8887a334d114680a5ab649b84f28fb63: Status 404 returned error can't find the container with id 104f1869a31d0e686f4c19a918f0d93a8887a334d114680a5ab649b84f28fb63 Oct 30 15:38:06.972609 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:06.972567 2607 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c99x2_0addea81-c333-4eed-8afa-3762350172bc/node-ca/0.log" Oct 30 15:38:07.168951 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:07.168871 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tcrzp" event={"ID":"4581006f-4626-475a-9942-da3b81d5d93b","Type":"ContainerStarted","Data":"104f1869a31d0e686f4c19a918f0d93a8887a334d114680a5ab649b84f28fb63"} Oct 30 15:38:08.175524 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:08.175432 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j24jl" event={"ID":"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171","Type":"ContainerStarted","Data":"a219837582f03a05c26e86917078cab379cb3546c5a4a1a979eb800ee0d402f7"} Oct 30 15:38:08.175524 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:08.175496 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j24jl" event={"ID":"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171","Type":"ContainerStarted","Data":"5574f3643835e273f55b61ebde8349eb5e5e926fdc83b7d492dac1e8d92c269c"} Oct 30 15:38:08.224335 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:08.224275 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-j24jl" podStartSLOduration=34.912416215 podStartE2EDuration="36.224256326s" totalImagesPullingTime="1.311840111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:08.223660503 +0000 UTC m=+36.873887786" watchObservedRunningTime="2026-10-30 15:38:08.224256326 +0000 UTC m=+36.874483611" Oct 30 15:38:10.182381 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:10.181937 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tcrzp" event={"ID":"4581006f-4626-475a-9942-da3b81d5d93b","Type":"ContainerStarted","Data":"74eea7304168cb5c0f9c0c0a5e682a8e7cae6ea628efc1a9268eb00600b9312b"} Oct 30 15:38:10.182381 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:10.182074 2607 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:10.227617 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:10.227565 2607 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tcrzp" podStartSLOduration=34.623511252 podStartE2EDuration="38.227549874s" totalImagesPullingTime="3.604038622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:10.226661904 +0000 UTC m=+38.876889194" watchObservedRunningTime="2026-10-30 15:38:10.227549874 +0000 UTC m=+38.877777158" Oct 30 15:38:11.186087 ip-10-0-133-222 kubenswrapper[2607]: I1030 15:38:11.186041 2607 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8r6df" event={"ID":"5a9c1136-39de-45c4-ba36-d8a4b524f252","Type":"ContainerStarted","Data":"062db3ad9e725a347a8e3490cbdc43bb1c8f48e8164f14600fa63049074ef2c8"} Oct 30 15:38:11.250725 ip-10-0-133-222 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 15:38:11.261244 ip-10-0-133-222 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 15:38:11.261401 ip-10-0-133-222 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 15:38:11.261450 ip-10-0-133-222 systemd[1]: kubelet.service: Consumed 2.104s CPU time. Oct 30 15:38:11.272330 ip-10-0-133-222 systemd[1]: Starting Kubernetes Kubelet... Oct 30 15:38:11.345246 ip-10-0-133-222 kubenswrapper[4401]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 15:38:11.345246 ip-10-0-133-222 kubenswrapper[4401]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 15:38:11.345246 ip-10-0-133-222 kubenswrapper[4401]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 15:38:11.345246 ip-10-0-133-222 kubenswrapper[4401]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 15:38:11.347268 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347196 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.347268 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347268 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.347268 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347271 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347274 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347277 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347281 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347284 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347287 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347290 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347292 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347295 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347298 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347301 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347304 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347306 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347309 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347312 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347315 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347318 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347320 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347323 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347326 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.347401 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347329 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347331 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347334 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347337 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347340 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347343 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347346 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347349 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347351 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347354 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347357 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347359 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347362 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347366 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347369 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347371 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347374 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347377 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347380 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347383 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.347918 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347385 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347388 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347391 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347394 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347396 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347399 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347401 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347405 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347407 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347411 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347416 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347420 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347423 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347427 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347430 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347432 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347435 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347438 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347440 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.348409 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347443 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347446 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347449 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347452 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347454 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347457 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347460 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347462 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347465 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347468 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347470 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347473 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347475 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347478 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347481 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347484 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347487 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347490 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347492 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.348886 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347495 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347498 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347500 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347503 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347506 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347509 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347512 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347514 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347518 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347520 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347523 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347527 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347531 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347534 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347537 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347541 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347543 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347546 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347549 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.349351 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347551 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347554 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347558 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347560 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347563 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347566 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.347569 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348320 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348327 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348332 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348336 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348339 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348342 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348345 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348348 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348350 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348353 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348357 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348359 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.349839 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348362 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348365 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348368 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348371 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348374 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348376 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348379 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348382 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348384 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348388 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348391 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348393 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348396 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348399 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348401 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348404 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348407 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348410 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348412 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348415 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.350313 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348418 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348421 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348424 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348426 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348429 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348432 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348434 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348437 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348440 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348442 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348445 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348448 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348450 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348453 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348455 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348458 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348461 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348464 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348467 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.350805 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348470 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348473 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348476 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348478 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348481 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348484 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348486 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348489 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348491 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348494 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348497 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348499 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348502 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348504 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348507 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348510 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348513 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348515 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348518 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348521 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.351303 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348523 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348526 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348531 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348534 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348537 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348539 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348542 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348545 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348547 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348550 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348553 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348556 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348559 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348562 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348565 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348567 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348570 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348573 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348576 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.351800 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348578 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348581 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348583 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348586 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348589 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348592 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348594 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348597 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348600 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348603 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348605 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348608 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348610 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348613 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348616 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.348620 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349175 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349181 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349185 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349190 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.352271 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349193 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349196 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349199 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349202 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349205 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349207 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349210 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349213 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349215 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349220 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349224 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349227 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349230 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349233 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349235 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349238 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349241 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349244 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349246 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.352840 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349249 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349254 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349257 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349260 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349262 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349265 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349268 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349270 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349273 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349276 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349278 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349281 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349284 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349287 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349290 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349293 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349295 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349298 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349301 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349303 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.353322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349306 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349308 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349311 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349315 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349318 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349321 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349324 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349327 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349330 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349334 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349337 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349340 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349343 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349346 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349349 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349352 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349355 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349358 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349361 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349364 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.353803 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349367 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349369 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349372 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349375 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349377 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349380 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349383 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349385 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349388 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349390 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349394 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349396 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349399 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349402 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349405 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349407 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349410 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349412 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349415 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349418 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.354312 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349420 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349424 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349426 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349429 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349431 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349435 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349438 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349440 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349443 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349446 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349448 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349451 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349454 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349456 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349459 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349461 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349464 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349467 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349470 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349472 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.354802 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349475 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349477 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.349480 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349564 4401 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349571 4401 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349578 4401 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349584 4401 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349590 4401 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349595 4401 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349602 4401 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349607 4401 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349611 4401 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349614 4401 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349618 4401 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349621 4401 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349624 4401 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349628 4401 flags.go:64] FLAG: --cgroup-root="" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349632 4401 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349635 4401 flags.go:64] FLAG: --client-ca-file="" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349639 4401 flags.go:64] FLAG: --cloud-provider="external" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349642 4401 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349646 4401 flags.go:64] FLAG: --cluster-domain="" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349650 4401 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349653 4401 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 15:38:11.355642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349656 4401 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349660 4401 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349664 4401 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349667 4401 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349670 4401 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349673 4401 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349676 4401 flags.go:64] FLAG: --contention-profiling="false" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349680 4401 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349683 4401 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349686 4401 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349689 4401 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349693 4401 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349697 4401 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349700 4401 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349703 4401 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349706 4401 flags.go:64] FLAG: --enable-server="true" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349709 4401 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349713 4401 flags.go:64] FLAG: --event-burst="100" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349716 4401 flags.go:64] FLAG: --event-qps="50" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349719 4401 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349723 4401 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349726 4401 flags.go:64] FLAG: --eviction-hard="" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349730 4401 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349733 4401 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349736 4401 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 15:38:11.356319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349740 4401 flags.go:64] FLAG: --eviction-soft="" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349743 4401 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349746 4401 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349749 4401 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349752 4401 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349755 4401 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349758 4401 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349761 4401 flags.go:64] FLAG: --feature-gates="" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349765 4401 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349768 4401 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349771 4401 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349774 4401 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349777 4401 flags.go:64] FLAG: --healthz-port="10248" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349780 4401 flags.go:64] FLAG: --help="false" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349783 4401 flags.go:64] FLAG: --hostname-override="ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349786 4401 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349789 4401 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349792 4401 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349795 4401 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349799 4401 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349802 4401 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349805 4401 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349808 4401 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 15:38:11.356972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349811 4401 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349814 4401 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349817 4401 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349831 4401 flags.go:64] FLAG: --kube-reserved="" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349834 4401 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349837 4401 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349840 4401 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349843 4401 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349846 4401 flags.go:64] FLAG: --lock-file="" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349850 4401 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349853 4401 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349857 4401 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349863 4401 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349866 4401 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349869 4401 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349872 4401 flags.go:64] FLAG: --logging-format="text" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349875 4401 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349878 4401 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349881 4401 flags.go:64] FLAG: --manifest-url="" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349884 4401 flags.go:64] FLAG: --manifest-url-header="" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349888 4401 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349891 4401 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349895 4401 flags.go:64] FLAG: --max-pods="110" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349898 4401 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349901 4401 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 15:38:11.357532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349904 4401 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349906 4401 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349909 4401 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349914 4401 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349917 4401 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349924 4401 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349927 4401 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349930 4401 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349933 4401 flags.go:64] FLAG: --pod-cidr="" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349936 4401 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349939 4401 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349942 4401 flags.go:64] FLAG: --pods-per-core="0" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349944 4401 flags.go:64] FLAG: --port="10250" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349948 4401 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349950 4401 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0949fc1c082626899" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349954 4401 flags.go:64] FLAG: --qos-reserved="" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349958 4401 flags.go:64] FLAG: --read-only-port="10255" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349961 4401 flags.go:64] FLAG: --register-node="true" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349964 4401 flags.go:64] FLAG: --register-with-taints="" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349968 4401 flags.go:64] FLAG: --registry-burst="10" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349970 4401 flags.go:64] FLAG: --registry-qps="5" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349974 4401 flags.go:64] FLAG: --reserved-cpus="" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349977 4401 flags.go:64] FLAG: --reserved-memory="" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349981 4401 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349984 4401 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349987 4401 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 15:38:11.358205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349990 4401 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349993 4401 flags.go:64] FLAG: --runonce="false" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349996 4401 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.349999 4401 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350002 4401 flags.go:64] FLAG: --seccomp-default="false" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350005 4401 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350007 4401 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350011 4401 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350014 4401 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350018 4401 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350021 4401 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350024 4401 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350027 4401 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350030 4401 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350033 4401 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350036 4401 flags.go:64] FLAG: --system-cgroups="" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350039 4401 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350044 4401 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350047 4401 flags.go:64] FLAG: --tls-cert-file="" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350050 4401 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350054 4401 flags.go:64] FLAG: --tls-min-version="" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350057 4401 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350065 4401 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350068 4401 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350071 4401 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 15:38:11.358836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350074 4401 flags.go:64] FLAG: --v="2" Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350078 4401 flags.go:64] FLAG: --version="false" Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350082 4401 flags.go:64] FLAG: --vmodule="" Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350086 4401 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350089 4401 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350212 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350215 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350218 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350222 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350224 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350227 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350230 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350232 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350235 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350246 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350249 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350254 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350256 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350259 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350279 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.359468 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350295 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350299 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350303 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350305 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350308 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350311 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350314 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350317 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350320 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350324 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350327 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350330 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350333 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350335 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350338 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350341 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350344 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350346 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350349 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.359969 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350352 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350355 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350357 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350360 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350363 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350366 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350368 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350371 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350374 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350378 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350380 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350383 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350386 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350388 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350391 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350394 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350397 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350400 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350402 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350405 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.360533 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350407 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350410 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350414 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350417 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350419 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350422 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350425 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350428 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350430 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350433 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350436 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350438 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350441 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350444 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350446 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350449 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350452 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350454 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350457 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.361322 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350461 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350466 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350471 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350475 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350479 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350482 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350485 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350488 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350491 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350494 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350497 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350500 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350503 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350505 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350508 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350512 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350515 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350518 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350520 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.362008 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350523 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350526 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350528 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350531 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350534 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350537 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350539 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350542 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350544 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350547 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350549 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350552 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350555 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350557 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.350602 4401 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350636 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.362556 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350644 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350647 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350649 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350652 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350655 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350658 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350661 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350664 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350667 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350669 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350672 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350674 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350679 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350682 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350684 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350687 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350690 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350692 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350695 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350698 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.362967 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350701 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350703 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350706 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350709 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350711 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350714 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350717 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350719 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350722 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350725 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350728 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350730 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350734 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350737 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350739 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350742 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350744 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350747 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350749 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350752 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.363594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350755 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350758 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350760 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350763 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350766 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350769 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350772 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350774 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350777 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350780 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350783 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350785 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350788 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350791 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350793 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350796 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350799 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350802 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350805 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350807 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.364202 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350810 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350813 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350816 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350818 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350836 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350839 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350842 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350844 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350848 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350852 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350855 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350859 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350864 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350867 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350870 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350873 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350876 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350879 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350882 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.364996 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350884 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350887 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350889 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350892 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350895 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350897 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350900 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350903 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350906 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350908 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350911 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350914 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350916 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350919 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350921 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350924 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350927 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350931 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350933 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350936 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.366025 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350939 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350942 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350944 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350947 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350950 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.350954 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.356298 4401 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.356313 4401 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.356337 4401 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.356343 4401 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356395 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356399 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356403 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356407 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356410 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356414 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356419 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356422 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356435 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.366881 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356438 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356441 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356444 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356447 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356450 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356453 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356456 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356458 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356461 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356464 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356468 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356471 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356474 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356476 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356479 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356482 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356484 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356487 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356490 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356493 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.367689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356496 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356498 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356501 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356504 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356507 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356509 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356512 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356514 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356517 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356520 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356522 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356525 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356528 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356530 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356533 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356535 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356538 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356541 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356544 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356547 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.368558 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356549 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356552 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356555 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356558 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356560 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356563 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356565 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356568 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356571 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356574 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356576 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356579 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356581 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356584 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356587 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356589 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356592 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356594 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356597 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.369407 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356601 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356605 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356609 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356611 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356614 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356617 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356619 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356622 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356625 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356628 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356631 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356635 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356638 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356640 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356643 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356646 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356649 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356651 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356654 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.370285 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356656 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356659 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356662 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356664 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356667 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356669 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356672 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356675 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356678 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356680 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356683 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356685 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356688 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356691 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356693 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356696 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356698 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356701 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.371262 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356703 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.356731 4401 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356888 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356893 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356896 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356899 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356902 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356906 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356909 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356912 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356915 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356917 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356920 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356923 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356926 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356928 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.372092 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356931 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356934 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356936 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356939 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356942 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356944 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356947 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356950 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356952 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356955 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356958 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356960 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356964 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356968 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356972 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356975 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356978 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356981 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356983 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.372812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356987 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356989 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356992 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356995 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.356997 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357001 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357004 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357007 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357011 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357014 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357017 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357020 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357023 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357026 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357029 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357032 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357034 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357037 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357040 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.373698 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357043 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357045 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357048 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357051 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357054 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357056 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357059 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357062 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357064 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357067 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357069 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357072 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357075 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357078 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357081 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357083 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357086 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357089 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357092 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.374458 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357095 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357098 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357100 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357103 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357106 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357108 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357111 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357114 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357116 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357119 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357121 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357124 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357127 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357129 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357132 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357135 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357137 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357140 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357142 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357145 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.375390 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357147 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357150 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357153 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357156 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357159 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357162 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357165 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357167 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357170 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357175 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357178 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357180 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357183 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357186 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357188 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.357212 4401 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 15:38:11.376035 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357242 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357246 4401 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357249 4401 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357253 4401 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357256 4401 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357259 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357262 4401 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357265 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357269 4401 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357272 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357274 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357277 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357280 4401 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357282 4401 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357285 4401 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357288 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357291 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357293 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357296 4401 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 15:38:11.376420 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357298 4401 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357301 4401 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357304 4401 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357306 4401 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357309 4401 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357311 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357316 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357319 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357322 4401 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357325 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357327 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357331 4401 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357333 4401 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357336 4401 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357338 4401 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357341 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357344 4401 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357347 4401 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357350 4401 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357352 4401 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 15:38:11.377238 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357355 4401 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357360 4401 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357362 4401 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357365 4401 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357368 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357370 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357373 4401 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357375 4401 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357378 4401 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357381 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357383 4401 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357386 4401 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357389 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357391 4401 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357394 4401 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357397 4401 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357399 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357402 4401 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357406 4401 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357409 4401 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 15:38:11.378032 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357412 4401 feature_gate.go:386] unrecognized feature gate: Example Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357415 4401 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357417 4401 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357420 4401 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357423 4401 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357426 4401 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357430 4401 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357432 4401 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357435 4401 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357438 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357440 4401 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357443 4401 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357445 4401 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357450 4401 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357454 4401 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357458 4401 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357461 4401 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357464 4401 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357467 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 15:38:11.378878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357470 4401 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357473 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357475 4401 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357478 4401 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357481 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357483 4401 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357486 4401 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357489 4401 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357492 4401 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357494 4401 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357498 4401 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357503 4401 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357506 4401 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357509 4401 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357511 4401 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357514 4401 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357517 4401 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357519 4401 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357522 4401 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 15:38:11.379381 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357525 4401 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357528 4401 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357531 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357534 4401 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357537 4401 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357540 4401 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357542 4401 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357547 4401 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:11.357550 4401 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.357601 4401 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.358564 4401 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.358621 4401 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.358798 4401 server.go:1014] "Starting client certificate rotation" Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.358923 4401 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.358969 4401 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 15:32:31 +0000 UTC" deadline="2028-03-26 17:33:37.414261156 +0000 UTC" Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.359005 4401 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12313h55m26.055258107s" Oct 30 15:38:11.379983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.359284 4401 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 15:38:11.380369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.360328 4401 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 15:38:11.380369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.361792 4401 log.go:25] "Validated CRI v1 runtime API" Oct 30 15:38:11.380369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.363253 4401 log.go:25] "Validated CRI v1 image API" Oct 30 15:38:11.380369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.363935 4401 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 15:38:11.380369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.365802 4401 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f4778c6f-330e-4507-96c0-0d63a0ac521d:/dev/nvme0n1p3 fec848c1-aa96-410d-bbed-8477a8f7db2c:/dev/nvme0n1p4] Oct 30 15:38:11.380537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.365845 4401 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/104f1869a31d0e686f4c19a918f0d93a8887a334d114680a5ab649b84f28fb63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/104f1869a31d0e686f4c19a918f0d93a8887a334d114680a5ab649b84f28fb63/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/19e38ef81b3fd35a26dca2df79bd7e7b723860f6254d983baaad45afdcea253e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/19e38ef81b3fd35a26dca2df79bd7e7b723860f6254d983baaad45afdcea253e/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28ad7854edbf25c99b0b385639f98172aa0d1023541e4912959581c22ab73454/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28ad7854edbf25c99b0b385639f98172aa0d1023541e4912959581c22ab73454/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b48ee46f793ce7b9d046ff088609679e18db41d13c10946a887cb0c856a8cb1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b48ee46f793ce7b9d046ff088609679e18db41d13c10946a887cb0c856a8cb1/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40743ccae829831cc26adc1b80a6ba2ff49ee6c1991c3b786cf3724e54ba8647/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40743ccae829831cc26adc1b80a6ba2ff49ee6c1991c3b786cf3724e54ba8647/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45d9cc5ff6a236f9d4f67746ad303844fddb6388eafe9930f6eb134c66666947/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45d9cc5ff6a236f9d4f67746ad303844fddb6388eafe9930f6eb134c66666947/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/577e4ca0aa157429408ed80b8d5ccec4cd9c261ed056554926b52346cbd7d77b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/577e4ca0aa157429408ed80b8d5ccec4cd9c261ed056554926b52346cbd7d77b/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6dc9934ae87fb3d89a43ae47192a7a3ecc0d172429002481617440605b8d2d45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6dc9934ae87fb3d89a43ae47192a7a3ecc0d172429002481617440605b8d2d45/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dc7d8257e7a660bd02a1bf20250db5c4a13152bcc6c3c7b46954361d49b7dae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dc7d8257e7a660bd02a1bf20250db5c4a13152bcc6c3c7b46954361d49b7dae/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b82a9fb94dcfe4e5a0cc1292dd1fed24a03311f1d4839eefbefe96cf2681c332/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b82a9fb94dcfe4e5a0cc1292dd1fed24a03311f1d4839eefbefe96cf2681c332/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e743ea90bf277a5dd9ebcaafa6f9fd870a95126799d362b9c586de73e0668c31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e743ea90bf277a5dd9ebcaafa6f9fd870a95126799d362b9c586de73e0668c31/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0addea81-c333-4eed-8afa-3762350172bc/volumes/kubernetes.io~projected/kube-api-access-cglh8:{mountpoint:/var/lib/kubelet/pods/0addea81-c333-4eed-8afa-3762350172bc/volumes/kubernetes.io~projected/kube-api-access-cglh8 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~projected/kube-api-access-z8wg4:{mountpoint:/var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~projected/kube-api-access-z8wg4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949/volumes/kubernetes.io~projected/kube-api-access-ffz48:{mountpoint:/var/lib/kubelet/pods/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949/volumes/kubernetes.io~projected/kube-api-access-ffz48 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/395236fd-641c-4bd3-8f77-3e0ac6cd29c0/volumes/kubernetes.io~projected/kube-api-access-2g9dm:{mountpoint:/var/lib/kubelet/pods/395236fd-641c-4bd3-8f77-3e0ac6cd29c0/volumes/kubernetes.io~projected/kube-api-access-2g9dm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4581006f-4626-475a-9942-da3b81d5d93b/volumes/kubernetes.io~projected/kube-api-access-86d7k:{mountpoint:/var/lib/kubelet/pods/4581006f-4626-475a-9942-da3b81d5d93b/volumes/kubernetes.io~projected/kube-api-access-86d7k major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171/volumes/kubernetes.io~projected/kube-api-access-76t8z:{mountpoint:/var/lib/kubelet/pods/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171/volumes/kubernetes.io~projected/kube-api-access-76t8z major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171/volumes/kubernetes.io~secret/metrics-certs major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a9c1136-39de-45c4-ba36-d8a4b524f252/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5a9c1136-39de-45c4-ba36-d8a4b524f252/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f01c529-80bc-4609-a6dd-912a16dfbba8/volumes/kubernetes.io~projected/kube-api-access-xkjsx:{mountpoint:/var/lib/kubelet/pods/7f01c529-80bc-4609-a6dd-912a16dfbba8/volumes/kubernetes.io~projected/kube-api-access-xkjsx major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volumes/kubernetes.io~projected/kube-api-access-67s8z:{mountpoint:/var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volumes/kubernetes.io~projected/kube-api-access-67s8z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da6d7847-fb28-415a-a5fb-765e87929cd4/volumes/kubernetes.io~projected/kube-api-access-c5qwr:{mountpoint:/var/lib/kubelet/pods/da6d7847-fb28-415a-a5fb-765e87929cd4/volumes/kubernetes.io~projected/kube-api-access-c5qwr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f427009a-c18a-404f-9948-9453d93d132b/volumes/kubernetes.io~projected/kube-api-access-g5k9k:{mountpoint:/var/lib/kubelet/pods/f427009a-c18a-404f-9948-9453d93d132b/volumes/kubernetes.io~projected/kube-api-access-g5k9k major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0b18614e35c5007b76d3eb21fbdaf951537d4dc16960cc79d81fd17fb32e4325/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/478eccf0f95d36b7f99a5ab556d75aa2a4fabc797d4bdc3dbbba4eb1222864fe/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/66c3d046e96ee711e8965089a8f93089ae472904e64485b56d0036a3dc51e6b5/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4cd8c7e883703a480a303298aae03c26177fc34fe45c730872d4329b69a10403/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ec2126bf45566a001d35920be316d3711e186665adeab7da34d3f9a84f43df77/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5a09ae4e60bf155209a65052b4b55fdb15643bfd34134cf7c8f17a0812939bf8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/97fb59df4565e409b49ba40ea18318d2c272e060549379a81f4f0659cf962f4f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/92935864c2363fe7c9eb0ce2473632671745ccf4413ee80cd1e13a661012d4be/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/fd7e4e6b3673fa2182a09da38511348068d6df5e8f0f82f827984945f11d263a/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/3953ac5ddfbf3f6aca85ded9b3df32361945bc52246644bd853c9dc49b3a48a3/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/cf10346a405eb9febf537fb12b7e8d072a5c9a06b7cc31e39fff4b81abaf6602/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/97b465e53bdcdb7368499d98d8aaa3a8e664e28ca2741448ec3a6b7ba3bb0a36/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/d3054950d491b6684cc05f2cb97de00dd0363b7c6da033f338798255e6f4f7a5/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b9785b64942f7497f1ad2caa3568a573e63de34496e44fc339628251a0b2ea95/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/00f0b314842e2a5e82401f5af8d28662d57b5e7b3fc3086f8f170f9f5589a437/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/61310c75b3cad45c950b5e96314fc75073a8a22e9b163cae8e4f12f270949c3b/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1fedbeb057ba166475ce8b785daedc56869fc22df0eee4c61ead2a8edfb35357/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/492449a9b1031226570adfc0872b18640500116ab80b1a4bb4237015acb878a2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c27d34bb15fcde68a0c1e1acf5f0b556d21b6b14900bf79543ec69254107a63a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/19c7e7d1896416c93ff07fbea2939e755753f54a3564a8334e9d1dbb2a798d0a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/b25d114a32cac0a7beea1216eaeb7185702b3a30073e43aad13721a534746de0/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-210:{mountpoint:/var/lib/containers/storage/overlay/fe2cd7ed8fc95225b73a9e33600c9aba7a3c6bbc1a4c2a0d877b225faf864461/merged major:0 minor:210 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/5052d39e9f38b672613e3708c71de507dd0ca9630c577e27ad29819f97a3df68/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/e863a464d1de57fd0564478fc29ee2700d2456826a3ca3b0fbc9ec124d3abf0a/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/b0d13eb319cc0563052aa89f9f6b04c25908f476412a8fa95778f69faa4b9c5f/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/f54c96bf37d93815cbc045031da333241cd38e8f38ccf355f8c115d3fc84afff/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/865c235d39d4e3f07654758942d5aebce7a303c076671e971f406057b5b81d31/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/dbb356618053a23b613aee2cace9a0723c5ca0a1abf92c25094e06e942a558f5/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/4b0b5c9c8e652901173a6bc0cef01eed08bdd8e05c38c4cad6aee2b3457cc8cd/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a11f7763c3dcdd4e9f05cd7f01c0b1c088643dbafd86f746159ab6344b333c59/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/fe3bb013c598565eb11d669987cdfd5353845ce7c542c6c50a98a0890777d68d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/d58fc7c096627593f2cd87e801c4b1818ad28d106f1010917ccd641f01664309/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/eddf3af5ea92d5a3403c91641c35005e2342166fd66bd16523f47490092415b0/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/53042133cb1f6480e80d5cd66aaaccfd7ace3ad101f4aeb8f0b88e027b45c11a/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2767a8d842c66f17d4a74592382de4a28ef2b78cfe80782d1c54b30d3ab27f18/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8e8f9493a3d610c9d3b7e40c581cdf9aa7b5c1bf7be014e5212f2f1048474aa7/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/dca49ef16b9228207800b9cea40178abfb2c60fc07435837147b70f7f8de212a/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/0b91c9f1ef654e6cd677dbac4dd939d1bada69586c6cc7b315d98c74f069888b/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 15:38:11.380537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.376153 4401 manager.go:217] Machine: {Timestamp:2026-10-30 15:38:11.375691545 +0000 UTC m=+0.074964693 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100478 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec212f5a4e80a5980ab14200e45be55b SystemUUID:ec212f5a-4e80-a598-0ab1-4200e45be55b BootID:7026c9de-7ad5-4f12-a749-88febb768551 Filesystems:[{Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dc7d8257e7a660bd02a1bf20250db5c4a13152bcc6c3c7b46954361d49b7dae/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/104f1869a31d0e686f4c19a918f0d93a8887a334d114680a5ab649b84f28fb63/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e743ea90bf277a5dd9ebcaafa6f9fd870a95126799d362b9c586de73e0668c31/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a9c1136-39de-45c4-ba36-d8a4b524f252/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da6d7847-fb28-415a-a5fb-765e87929cd4/volumes/kubernetes.io~projected/kube-api-access-c5qwr DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volumes/kubernetes.io~projected/kube-api-access-67s8z DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/395236fd-641c-4bd3-8f77-3e0ac6cd29c0/volumes/kubernetes.io~projected/kube-api-access-2g9dm DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f01c529-80bc-4609-a6dd-912a16dfbba8/volumes/kubernetes.io~projected/kube-api-access-xkjsx DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/19e38ef81b3fd35a26dca2df79bd7e7b723860f6254d983baaad45afdcea253e/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b82a9fb94dcfe4e5a0cc1292dd1fed24a03311f1d4839eefbefe96cf2681c332/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f427009a-c18a-404f-9948-9453d93d132b/volumes/kubernetes.io~projected/kube-api-access-g5k9k DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28ad7854edbf25c99b0b385639f98172aa0d1023541e4912959581c22ab73454/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45d9cc5ff6a236f9d4f67746ad303844fddb6388eafe9930f6eb134c66666947/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949/volumes/kubernetes.io~projected/kube-api-access-ffz48 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6dc9934ae87fb3d89a43ae47192a7a3ecc0d172429002481617440605b8d2d45/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/577e4ca0aa157429408ed80b8d5ccec4cd9c261ed056554926b52346cbd7d77b/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4581006f-4626-475a-9942-da3b81d5d93b/volumes/kubernetes.io~projected/kube-api-access-86d7k DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0addea81-c333-4eed-8afa-3762350172bc/volumes/kubernetes.io~projected/kube-api-access-cglh8 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171/volumes/kubernetes.io~projected/kube-api-access-76t8z DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b48ee46f793ce7b9d046ff088609679e18db41d13c10946a887cb0c856a8cb1/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a64acc66-316d-49f8-8659-a609dbcc6f63/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40743ccae829831cc26adc1b80a6ba2ff49ee6c1991c3b786cf3724e54ba8647/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-210 DeviceMajor:0 DeviceMinor:210 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6651f4-00bb-4e63-94c3-0e7884f24b4a/volumes/kubernetes.io~projected/kube-api-access-z8wg4 DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:104f1869a31d0e6 MacAddress:72:de:3f:9b:09:ce Speed:10000 Mtu:8901} {Name:2b48ee46f793ce7 MacAddress:3a:54:fd:89:99:ad Speed:10000 Mtu:8901} {Name:45d9cc5ff6a236f MacAddress:da:54:b0:2e:39:be Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d7:55:13:13 Speed:0 Mtu:9001} {Name:br-int MacAddress:ae:bc:6b:8d:d0:1b Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:d7:55:13:13 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:d6:f7:be:43:84:5e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4e:c7:88:21:38:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 15:38:11.380537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.376382 4401 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 15:38:11.380537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.376421 4401 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 15:38:11.380537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.376805 4401 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.376854 4401 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-222.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377041 4401 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377052 4401 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377066 4401 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377084 4401 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377176 4401 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377416 4401 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377555 4401 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377599 4401 kubelet.go:493] "Attempting to sync node with API server" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377624 4401 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377637 4401 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377649 4401 kubelet.go:405] "Adding apiserver pod source" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.377661 4401 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.378538 4401 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.378552 4401 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.379066 4401 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.380911 4401 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 15:38:11.380918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.380920 4401 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381006 4401 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381173 4401 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381199 4401 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381280 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381291 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381297 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381302 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381309 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381317 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381324 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381334 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381344 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381354 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381365 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 15:38:11.381392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381376 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 15:38:11.381926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381409 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 15:38:11.381926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.381415 4401 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 15:38:11.384672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.384659 4401 server.go:1263] "Started kubelet" Oct 30 15:38:11.384792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.384752 4401 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 15:38:11.386063 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.385943 4401 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 15:38:11.386164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.386089 4401 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 15:38:11.386394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.386377 4401 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 15:38:11.386493 ip-10-0-133-222 systemd[1]: Started Kubernetes Kubelet. Oct 30 15:38:11.387660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.386782 4401 server.go:317] "Adding debug handlers to kubelet server" Oct 30 15:38:11.388107 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.388093 4401 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 15:38:11.388170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.388141 4401 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.390735 4401 factory.go:55] Registering systemd factory Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.390759 4401 factory.go:223] Registration of the systemd container factory successfully Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:11.390948 4401 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-222.ec2.internal\" not found" Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.391032 4401 factory.go:153] Registering CRI-O factory Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.391044 4401 factory.go:223] Registration of the crio container factory successfully Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.391101 4401 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.391130 4401 factory.go:103] Registering Raw factory Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.391147 4401 manager.go:1199] Started watching for new ooms in manager Oct 30 15:38:11.391859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.391559 4401 manager.go:319] Starting recovery of all containers Oct 30 15:38:11.393754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.393388 4401 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 15:38:11.393754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.393418 4401 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 15:38:11.393754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.393594 4401 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 15:38:11.393754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.393579 4401 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 15:32:33 +0000 UTC" deadline="2028-04-10 19:45:31.136674611 +0000 UTC" Oct 30 15:38:11.393754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.393647 4401 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12676h7m19.74303512s" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396439 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6651f4-00bb-4e63-94c3-0e7884f24b4a" volumeName="kubernetes.io/projected/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-kube-api-access-z8wg4" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396475 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4581006f-4626-475a-9942-da3b81d5d93b" volumeName="kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396490 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" volumeName="kubernetes.io/projected/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-kube-api-access-76t8z" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396504 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46e6b33b-39c7-4f7d-a3ba-4ce7c4337171" volumeName="kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396516 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64acc66-316d-49f8-8659-a609dbcc6f63" volumeName="kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-script-lib" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396531 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da6d7847-fb28-415a-a5fb-765e87929cd4" volumeName="kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-cni-binary-copy" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396549 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da6d7847-fb28-415a-a5fb-765e87929cd4" volumeName="kubernetes.io/projected/da6d7847-fb28-415a-a5fb-765e87929cd4-kube-api-access-c5qwr" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396563 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6651f4-00bb-4e63-94c3-0e7884f24b4a" volumeName="kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-tmp" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396578 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d9f9bb-0f4f-4110-9ad8-28ac0bbde949" volumeName="kubernetes.io/empty-dir/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-tmp-dir" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396590 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07" volumeName="kubernetes.io/configmap/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-konnectivity-ca" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396615 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07" volumeName="kubernetes.io/secret/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-agent-certs" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396626 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da6d7847-fb28-415a-a5fb-765e87929cd4" volumeName="kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-daemon-config" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396639 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6651f4-00bb-4e63-94c3-0e7884f24b4a" volumeName="kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-tuned" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396649 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395236fd-641c-4bd3-8f77-3e0ac6cd29c0" volumeName="kubernetes.io/projected/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-kube-api-access-2g9dm" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396664 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64acc66-316d-49f8-8659-a609dbcc6f63" volumeName="kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-config" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396675 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f427009a-c18a-404f-9948-9453d93d132b" volumeName="kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-binary-copy" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396687 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f427009a-c18a-404f-9948-9453d93d132b" volumeName="kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396707 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f427009a-c18a-404f-9948-9453d93d132b" volumeName="kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396720 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f427009a-c18a-404f-9948-9453d93d132b" volumeName="kubernetes.io/projected/f427009a-c18a-404f-9948-9453d93d132b-kube-api-access-g5k9k" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396732 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0addea81-c333-4eed-8afa-3762350172bc" volumeName="kubernetes.io/projected/0addea81-c333-4eed-8afa-3762350172bc-kube-api-access-cglh8" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396744 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31d9f9bb-0f4f-4110-9ad8-28ac0bbde949" volumeName="kubernetes.io/projected/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-kube-api-access-ffz48" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396756 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="395236fd-641c-4bd3-8f77-3e0ac6cd29c0" volumeName="kubernetes.io/configmap/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-iptables-alerter-script" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396769 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a9c1136-39de-45c4-ba36-d8a4b524f252" volumeName="kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396782 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f01c529-80bc-4609-a6dd-912a16dfbba8" volumeName="kubernetes.io/projected/7f01c529-80bc-4609-a6dd-912a16dfbba8-kube-api-access-xkjsx" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396793 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64acc66-316d-49f8-8659-a609dbcc6f63" volumeName="kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-env-overrides" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396804 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64acc66-316d-49f8-8659-a609dbcc6f63" volumeName="kubernetes.io/projected/a64acc66-316d-49f8-8659-a609dbcc6f63-kube-api-access-67s8z" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396818 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a64acc66-316d-49f8-8659-a609dbcc6f63" volumeName="kubernetes.io/secret/a64acc66-316d-49f8-8659-a609dbcc6f63-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396850 4401 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0addea81-c333-4eed-8afa-3762350172bc" volumeName="kubernetes.io/configmap/0addea81-c333-4eed-8afa-3762350172bc-serviceca" seLinuxMountContext="" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396867 4401 reconstruct.go:97] "Volume reconstruction finished" Oct 30 15:38:11.397333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.396874 4401 reconciler.go:29] "Reconciler: start to sync state" Oct 30 15:38:11.418568 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.418541 4401 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 15:38:11.419876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.419857 4401 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 15:38:11.419876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.419878 4401 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 15:38:11.420008 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.419896 4401 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 15:38:11.420008 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:11.419943 4401 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 15:38:11.432752 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.432729 4401 generic.go:356] "Generic (PLEG): container finished" podID="9895a994f5b5cfb1039cd968cd927e03" containerID="0fc07cac45980a200ac4cbf0d51434c473c1c95d4d307c2683493aee8f335407" exitCode=0 Oct 30 15:38:11.441933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.441900 4401 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="b6de40bbd8e19b12c2bf59bf2e50e7a6ec10e4edcf019e8728595f8be4cc631b" exitCode=0 Oct 30 15:38:11.441933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.441931 4401 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="7a7989e4d01d471661699617f142eae3369d60cc6d66aa0d4a94fa0196b5311e" exitCode=0 Oct 30 15:38:11.442088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.441942 4401 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="91559b52c3b3a62bb00b00c72ec15d29d11968fb2102fe47b99701cf34806c9c" exitCode=0 Oct 30 15:38:11.442088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.441951 4401 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="0a42003726e627a21ee9295751c6cf53e800848c83a4750c5d3fd27ead4a295b" exitCode=0 Oct 30 15:38:11.442088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.441956 4401 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="e906f031d7cf286e6273e62e3f61c4e14259012ee1cb7935522810116ead5431" exitCode=0 Oct 30 15:38:11.442088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.441964 4401 generic.go:356] "Generic (PLEG): container finished" podID="f427009a-c18a-404f-9948-9453d93d132b" containerID="e226b1847904a4d9dbbc3b5bb64ed306e2a710d9c47b7ba6d6ffb3bb27d1a2f2" exitCode=0 Oct 30 15:38:11.448352 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.448336 4401 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:38:11.449398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.449374 4401 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 15:38:11.449481 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.449455 4401 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:38:11.460745 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.460728 4401 manager.go:324] Recovery completed Oct 30 15:38:11.464319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.464300 4401 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:38:11.464494 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.464477 4401 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 15:38:11.476593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476567 4401 cpu_manager.go:226] "Starting" policy="none" Oct 30 15:38:11.476593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476592 4401 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 15:38:11.476737 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476621 4401 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 15:38:11.476809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476795 4401 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 15:38:11.476865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476809 4401 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 15:38:11.476865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476841 4401 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 15:38:11.476865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476847 4401 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 15:38:11.476865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476852 4401 policy_none.go:50] "Start" Oct 30 15:38:11.476865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476861 4401 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 15:38:11.477006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476870 4401 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 15:38:11.477006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476989 4401 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 15:38:11.477006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.476996 4401 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 15:38:11.477006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.477001 4401 policy_none.go:44] "Start" Oct 30 15:38:11.480769 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.480746 4401 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 15:38:11.483308 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.483295 4401 manager.go:341] "Starting Device Plugin manager" Oct 30 15:38:11.483837 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:11.483323 4401 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 15:38:11.483909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.483841 4401 server.go:87] "Starting device plugin registration server" Oct 30 15:38:11.484115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.484102 4401 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 15:38:11.484187 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.484116 4401 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 15:38:11.484235 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.484218 4401 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 15:38:11.484365 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.484316 4401 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 15:38:11.484365 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.484362 4401 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 15:38:11.484365 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.484369 4401 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 15:38:11.484607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.484403 4401 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T15:38:11.484351314Z","UUID":"0bbd03a1-dded-4507-9f7c-693e511229e9","Handler":null,"Name":"","Endpoint":""} Oct 30 15:38:11.485498 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:11.485478 4401 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 15:38:11.485805 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.485787 4401 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 15:38:11.485920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.485813 4401 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 15:38:11.520114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520049 4401 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal"] Oct 30 15:38:11.520500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520460 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"9895a994f5b5cfb1039cd968cd927e03","Type":"ContainerStarted","Data":"ed82353a60a77c0259a7a20bbf808af36d9ed16a414acb05978c7096b9c928f1"} Oct 30 15:38:11.520534 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520521 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"9895a994f5b5cfb1039cd968cd927e03","Type":"ContainerDied","Data":"0fc07cac45980a200ac4cbf0d51434c473c1c95d4d307c2683493aee8f335407"} Oct 30 15:38:11.520563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520545 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" event={"ID":"9895a994f5b5cfb1039cd968cd927e03","Type":"ContainerStarted","Data":"46b97a703b56c47852615a60cd30bbaee38a33bb27f48a397d29acaefbc177f5"} Oct 30 15:38:11.520599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520562 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" event={"ID":"e2bab30e2cbbbd657a1d2e615d7ce89c","Type":"ContainerStarted","Data":"821d54b7e84961c16737a40ac2642c5b566da446cc753a8311ca1121b505f3e2"} Oct 30 15:38:11.520599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520581 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" event={"ID":"e2bab30e2cbbbd657a1d2e615d7ce89c","Type":"ContainerStarted","Data":"34c512b84f1c1a8f530ef63f188074f61fdaf71c5f3240db4240dd8946123510"} Oct 30 15:38:11.520744 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520734 4401 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.520814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.520800 4401 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.544287 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.544273 4401 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 15:38:11.544364 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:11.544311 4401 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.544874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.544861 4401 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 15:38:11.544923 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:11.544898 4401 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.585034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.585006 4401 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 15:38:11.586321 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.586300 4401 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientMemory" Oct 30 15:38:11.586411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.586333 4401 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 15:38:11.586411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.586344 4401 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-222.ec2.internal" event="NodeHasSufficientPID" Oct 30 15:38:11.586411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.586378 4401 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.624338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.624314 4401 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.624444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.624392 4401 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.650250 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.650221 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bab30e2cbbbd657a1d2e615d7ce89c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"e2bab30e2cbbbd657a1d2e615d7ce89c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.650339 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.650256 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.650339 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.650281 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.750575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.750514 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bab30e2cbbbd657a1d2e615d7ce89c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"e2bab30e2cbbbd657a1d2e615d7ce89c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.750575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.750555 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.750780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.750629 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.750780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.750644 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e2bab30e2cbbbd657a1d2e615d7ce89c-config\") pod \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" (UID: \"e2bab30e2cbbbd657a1d2e615d7ce89c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.750780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.750682 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:11.750780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:11.750719 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9895a994f5b5cfb1039cd968cd927e03-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" (UID: \"9895a994f5b5cfb1039cd968cd927e03\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:12.378335 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.378299 4401 apiserver.go:52] "Watching apiserver" Oct 30 15:38:12.413550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.413526 4401 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 15:38:12.413887 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.413870 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-wws5p","openshift-multus/network-metrics-daemon-j24jl","openshift-network-operator/iptables-alerter-ztxfj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb","openshift-multus/multus-additional-cni-plugins-974j2","kube-system/konnectivity-agent-scbpb","kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal","openshift-network-diagnostics/network-check-target-tcrzp","openshift-ovn-kubernetes/ovnkube-node-2vkxh","kube-system/global-pull-secret-syncer-8r6df","openshift-cluster-node-tuning-operator/tuned-v8qs2","openshift-dns/node-resolver-cm4sh","openshift-image-registry/node-ca-c99x2"] Oct 30 15:38:12.433258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.433229 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 15:38:12.433258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.433257 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 15:38:12.433429 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.432783 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 15:38:12.433429 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.433310 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.433517 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.433494 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.433562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.433520 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5dknm\"" Oct 30 15:38:12.433605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.433565 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 15:38:12.434021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.434001 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 15:38:12.444878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.444851 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.445447 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.445434 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 15:38:12.446067 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446038 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.446189 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446088 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.446189 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446087 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-xbwc7\"" Oct 30 15:38:12.446312 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446302 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 15:38:12.446355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446318 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 15:38:12.446355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446350 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.446444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446353 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.446444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446377 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.446444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446394 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.446444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446416 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.446444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446429 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8vjmf\"" Oct 30 15:38:12.446444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446438 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.446648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446472 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 15:38:12.446648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446531 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ftvkm\"" Oct 30 15:38:12.446648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446598 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 15:38:12.446648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446601 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fmlw7\"" Oct 30 15:38:12.446757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446654 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-nlpvl\"" Oct 30 15:38:12.446757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.446734 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 15:38:12.447097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.447085 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 15:38:12.447302 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.447289 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-khtmm\"" Oct 30 15:38:12.447371 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.447359 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qtl5k\"" Oct 30 15:38:12.447680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.447664 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 15:38:12.447732 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.447691 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mw526\"" Oct 30 15:38:12.447775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.447759 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.447834 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.447794 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.448024 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448001 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-84s7k\"" Oct 30 15:38:12.448024 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448014 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.448024 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448007 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5ljdx\"" Oct 30 15:38:12.448177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448047 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 15:38:12.448177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448052 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 15:38:12.448177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448109 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ck89s\"" Oct 30 15:38:12.448177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448120 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 15:38:12.448177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.448154 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 15:38:12.455687 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.455672 4401 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:38:12.455852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.455838 4401 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:12.481791 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.481773 4401 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 15:38:12.481895 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:12.481811 4401 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-222.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-222.ec2.internal" Oct 30 15:38:12.482421 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.482404 4401 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 15:38:12.482468 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:12.482439 4401 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-222.ec2.internal" Oct 30 15:38:12.494592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.494575 4401 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 15:38:12.553402 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553380 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-systemd-units\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553405 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-netns\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553424 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-systemd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553442 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-script-lib\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553461 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67s8z\" (UniqueName: \"kubernetes.io/projected/a64acc66-316d-49f8-8659-a609dbcc6f63-kube-api-access-67s8z\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553612 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.553686 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553663 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5k9k\" (UniqueName: \"kubernetes.io/projected/f427009a-c18a-404f-9948-9453d93d132b-kube-api-access-g5k9k\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.553735 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553719 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-script-lib\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553797 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553782 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-host-slash\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:38:12.553851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553811 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-tuned\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.553885 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553866 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-tmp\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.553937 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553897 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553937 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553917 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-env-overrides\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.553937 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553930 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-tmp\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.554040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553930 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-tuned\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.554040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553957 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-cni-binary-copy\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.554040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553984 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-kubelet-config\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.554040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.553984 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.554040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554012 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-conf-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.554186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554113 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-daemon-config\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.554186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554137 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-env-overrides\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.554186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554140 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-modprobe-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.554186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554171 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-kubelet\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.554313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554251 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-cni-binary-copy\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.554313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554294 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-config\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.554313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554302 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-daemon-config\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.554427 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554340 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.554458 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554426 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0addea81-c333-4eed-8afa-3762350172bc-serviceca\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:38:12.554458 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554441 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:12.554520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554464 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a64acc66-316d-49f8-8659-a609dbcc6f63-ovnkube-config\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.554520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554474 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.554590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554570 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-systemd\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.554622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554595 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76t8z\" (UniqueName: \"kubernetes.io/projected/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-kube-api-access-76t8z\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:12.554650 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554630 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.554685 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554653 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-metrics-certs\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:12.554718 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554693 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:12.554757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554733 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-konnectivity-ca\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:38:12.554793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554761 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-bin\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.554793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554783 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-netd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.554871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554812 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cglh8\" (UniqueName: \"kubernetes.io/projected/0addea81-c333-4eed-8afa-3762350172bc-kube-api-access-cglh8\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:38:12.554871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554863 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-socket-dir-parent\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.554949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554884 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-cnibin\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.554949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554909 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-binary-copy\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.554949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554933 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-cnibin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.555037 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.554987 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-konnectivity-ca\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:38:12.555089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555071 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-etc-kubernetes\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.555125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555103 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-etc-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.555163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555125 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.555163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555137 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0addea81-c333-4eed-8afa-3762350172bc-serviceca\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:38:12.555219 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555161 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g9dm\" (UniqueName: \"kubernetes.io/projected/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-kube-api-access-2g9dm\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:38:12.555219 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555179 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f427009a-c18a-404f-9948-9453d93d132b-cni-binary-copy\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.555219 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555188 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-run\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555242 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.555313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555264 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.555313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555278 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-hosts-file\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:38:12.555313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555295 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-k8s-cni-cncf-io\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.555448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555345 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64acc66-316d-49f8-8659-a609dbcc6f63-ovn-node-metrics-cert\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.555448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555362 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjsx\" (UniqueName: \"kubernetes.io/projected/7f01c529-80bc-4609-a6dd-912a16dfbba8-kube-api-access-xkjsx\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.555448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555382 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-netns\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.555448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555405 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-var-lib-kubelet\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555442 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-var-lib-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.555614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555456 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.555614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555475 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysconfig\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555526 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555567 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-agent-certs\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:38:12.555614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555595 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-dbus\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.555614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555611 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-system-cni-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.555795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555629 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5qwr\" (UniqueName: \"kubernetes.io/projected/da6d7847-fb28-415a-a5fb-765e87929cd4-kube-api-access-c5qwr\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.555795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555650 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-iptables-alerter-script\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:38:12.555795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555728 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-sys\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555752 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-lib-modules\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555749 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a64acc66-316d-49f8-8659-a609dbcc6f63-ovn-node-metrics-cert\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555812 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-bin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555861 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07-agent-certs\") pod \"konnectivity-agent-scbpb\" (UID: \"3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07\") " pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555860 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-multus-certs\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555880 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-iptables-alerter-script\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555898 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0addea81-c333-4eed-8afa-3762350172bc-host\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555925 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-kubernetes\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555944 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-host\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.555993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555967 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-slash\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.555994 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-log-socket\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556028 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556056 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-tmp-dir\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556103 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-kubelet\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556119 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-tmp-dir\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556132 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffz48\" (UniqueName: \"kubernetes.io/projected/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-kube-api-access-ffz48\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556152 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-conf\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556167 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8wg4\" (UniqueName: \"kubernetes.io/projected/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-kube-api-access-z8wg4\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.556206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556188 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-ovn\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556294 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556343 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-device-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556375 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-system-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556396 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556434 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-multus\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556449 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-hostroot\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556462 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-node-log\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556478 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.556511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556493 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.556753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.556708 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5a9c1136-39de-45c4-ba36-d8a4b524f252-original-pull-secret\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.576835 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.576812 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5k9k\" (UniqueName: \"kubernetes.io/projected/f427009a-c18a-404f-9948-9453d93d132b-kube-api-access-g5k9k\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.578410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.578395 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67s8z\" (UniqueName: \"kubernetes.io/projected/a64acc66-316d-49f8-8659-a609dbcc6f63-kube-api-access-67s8z\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.578734 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.578718 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g9dm\" (UniqueName: \"kubernetes.io/projected/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-kube-api-access-2g9dm\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:38:12.580127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.580110 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cglh8\" (UniqueName: \"kubernetes.io/projected/0addea81-c333-4eed-8afa-3762350172bc-kube-api-access-cglh8\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:38:12.580601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.580583 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86d7k\" (UniqueName: \"kubernetes.io/projected/4581006f-4626-475a-9942-da3b81d5d93b-kube-api-access-86d7k\") pod \"network-check-target-tcrzp\" (UID: \"4581006f-4626-475a-9942-da3b81d5d93b\") " pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:12.584547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.584531 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76t8z\" (UniqueName: \"kubernetes.io/projected/46e6b33b-39c7-4f7d-a3ba-4ce7c4337171-kube-api-access-76t8z\") pod \"network-metrics-daemon-j24jl\" (UID: \"46e6b33b-39c7-4f7d-a3ba-4ce7c4337171\") " pod="openshift-multus/network-metrics-daemon-j24jl" Oct 30 15:38:12.592291 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.592270 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5qwr\" (UniqueName: \"kubernetes.io/projected/da6d7847-fb28-415a-a5fb-765e87929cd4-kube-api-access-c5qwr\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.592333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.592304 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8wg4\" (UniqueName: \"kubernetes.io/projected/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-kube-api-access-z8wg4\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.592460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.592445 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffz48\" (UniqueName: \"kubernetes.io/projected/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-kube-api-access-ffz48\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:38:12.592699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.592684 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjsx\" (UniqueName: \"kubernetes.io/projected/7f01c529-80bc-4609-a6dd-912a16dfbba8-kube-api-access-xkjsx\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.657388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657318 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-lib-modules\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.657388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657346 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-bin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.657388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657361 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-multus-certs\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.657388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657375 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0addea81-c333-4eed-8afa-3762350172bc-host\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:38:12.657388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657388 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-kubernetes\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657401 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-host\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657431 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-bin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657443 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0addea81-c333-4eed-8afa-3762350172bc-host\") pod \"node-ca-c99x2\" (UID: \"0addea81-c333-4eed-8afa-3762350172bc\") " pod="openshift-image-registry/node-ca-c99x2" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657443 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-multus-certs\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657468 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-slash\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657453 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-host\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657482 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-kubernetes\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657498 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-log-socket\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657527 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-lib-modules\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657529 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657558 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-slash\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657581 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-log-socket\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657598 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-socket-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657608 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-kubelet\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657638 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-conf\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657661 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-kubelet\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657663 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-ovn\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.657697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657690 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657716 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-device-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657738 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-conf\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657743 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-ovn\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657738 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-system-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657769 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657781 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657785 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-device-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657794 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-system-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657798 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-multus\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657816 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-var-lib-cni-multus\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657839 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-sys-fs\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657855 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-hostroot\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657870 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-hostroot\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657883 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-node-log\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657909 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657931 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-systemd-units\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657938 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-node-log\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657959 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-netns\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657975 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-systemd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657978 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-systemd-units\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.657993 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-host-slash\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658015 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-cni-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658018 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658025 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/395236fd-641c-4bd3-8f77-3e0ac6cd29c0-host-slash\") pod \"iptables-alerter-ztxfj\" (UID: \"395236fd-641c-4bd3-8f77-3e0ac6cd29c0\") " pod="openshift-network-operator/iptables-alerter-ztxfj" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658020 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-netns\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658038 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-kubelet-config\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658022 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-systemd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658053 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658063 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-conf-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658066 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-kubelet-config\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658080 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-modprobe-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658107 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-kubelet\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658107 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-conf-dir\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658126 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658142 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-systemd\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658158 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-bin\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658168 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-kubelet\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658177 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-netd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658193 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-modprobe-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658178 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658198 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-bin\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658219 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-socket-dir-parent\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658194 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-systemd\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658236 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-cnibin\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658196 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-cni-netd\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658263 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-cnibin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658285 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-etc-kubernetes\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658296 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-cnibin\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658301 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-etc-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658304 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-cnibin\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658321 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-etc-kubernetes\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658318 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658337 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-etc-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658342 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658346 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-run\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658366 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658372 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-run\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658306 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-multus-socket-dir-parent\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658381 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658413 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-registration-dir\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658420 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f01c529-80bc-4609-a6dd-912a16dfbba8-etc-selinux\") pod \"aws-ebs-csi-driver-node-xzvfb\" (UID: \"7f01c529-80bc-4609-a6dd-912a16dfbba8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xzvfb" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658441 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-hosts-file\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:38:12.658430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658460 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-k8s-cni-cncf-io\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658476 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-netns\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658490 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-var-lib-kubelet\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658512 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-k8s-cni-cncf-io\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658513 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-var-lib-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658526 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/da6d7847-fb28-415a-a5fb-765e87929cd4-host-run-netns\") pod \"multus-wws5p\" (UID: \"da6d7847-fb28-415a-a5fb-765e87929cd4\") " pod="openshift-multus/multus-wws5p" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658560 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/31d9f9bb-0f4f-4110-9ad8-28ac0bbde949-hosts-file\") pod \"node-resolver-cm4sh\" (UID: \"31d9f9bb-0f4f-4110-9ad8-28ac0bbde949\") " pod="openshift-dns/node-resolver-cm4sh" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658592 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658608 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-var-lib-kubelet\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658607 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-var-lib-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658641 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysconfig\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658657 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658672 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-dbus\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658679 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a64acc66-316d-49f8-8659-a609dbcc6f63-run-openvswitch\") pod \"ovnkube-node-2vkxh\" (UID: \"a64acc66-316d-49f8-8659-a609dbcc6f63\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658687 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-system-cni-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658705 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-sys\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658714 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysconfig\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658743 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f427009a-c18a-404f-9948-9453d93d132b-system-cni-dir\") pod \"multus-additional-cni-plugins-974j2\" (UID: \"f427009a-c18a-404f-9948-9453d93d132b\") " pod="openshift-multus/multus-additional-cni-plugins-974j2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658768 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-sys\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658767 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5a9c1136-39de-45c4-ba36-d8a4b524f252-dbus\") pod \"global-pull-secret-syncer-8r6df\" (UID: \"5a9c1136-39de-45c4-ba36-d8a4b524f252\") " pod="kube-system/global-pull-secret-syncer-8r6df" Oct 30 15:38:12.660304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:12.658792 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6651f4-00bb-4e63-94c3-0e7884f24b4a-etc-sysctl-d\") pod \"tuned-v8qs2\" (UID: \"0b6651f4-00bb-4e63-94c3-0e7884f24b4a\") " pod="openshift-cluster-node-tuning-operator/tuned-v8qs2" Oct 30 15:38:13.543173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:13.543141 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:15.698570 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:15.698539 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-scbpb" Oct 30 15:38:18.388976 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.388943 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:18.389336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.389211 4401 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 15:38:18.511010 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.510980 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:18.746688 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.746624 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nrn5l"] Oct 30 15:38:18.749282 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.749258 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-krv4w"] Oct 30 15:38:18.749427 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.749402 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.750914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.750895 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:18.751442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.751417 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm"] Oct 30 15:38:18.752970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.752952 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" Oct 30 15:38:18.780212 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.780192 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 15:38:18.780212 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.780210 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 15:38:18.781034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.781013 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 15:38:18.781034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.781030 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 15:38:18.781231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.781057 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 15:38:18.785020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.784996 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 15:38:18.785147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.785030 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rf9nq\"" Oct 30 15:38:18.785147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.785136 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 15:38:18.785459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.785228 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-ws7hq\"" Oct 30 15:38:18.785459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.785300 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2xz8v\"" Oct 30 15:38:18.786769 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.786748 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vd8pb"] Oct 30 15:38:18.789572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.789552 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrn5l"] Oct 30 15:38:18.789675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.789651 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.790527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.790509 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-krv4w"] Oct 30 15:38:18.800497 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.800480 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9slvd"] Oct 30 15:38:18.801885 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.801872 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:18.806451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.806423 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm"] Oct 30 15:38:18.812906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.812888 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 15:38:18.816618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.816597 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vd8pb"] Oct 30 15:38:18.840343 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.840321 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9slvd"] Oct 30 15:38:18.851878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.851860 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 15:38:18.853207 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.853193 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 15:38:18.886073 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886047 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e61536a3-3592-485a-98bd-93cbae66c2cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rzswm\" (UID: \"e61536a3-3592-485a-98bd-93cbae66c2cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" Oct 30 15:38:18.886182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886077 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf1a0534-442c-4393-9f6e-09811a28e8c5-crio-socket\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.886182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886095 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d418480f-1d70-41ce-902f-c36c0c1b4b97-metrics-tls\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.886182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886115 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1a0534-442c-4393-9f6e-09811a28e8c5-data-volume\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.886182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886167 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf1a0534-442c-4393-9f6e-09811a28e8c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.886314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886247 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgdj\" (UniqueName: \"kubernetes.io/projected/cf1a0534-442c-4393-9f6e-09811a28e8c5-kube-api-access-zjgdj\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.886314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886273 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d418480f-1d70-41ce-902f-c36c0c1b4b97-config-volume\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.886375 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886337 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da25fd18-f13d-419a-aae5-f9a74ba5d045-cert\") pod \"ingress-canary-9slvd\" (UID: \"da25fd18-f13d-419a-aae5-f9a74ba5d045\") " pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:18.886438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886419 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf1a0534-442c-4393-9f6e-09811a28e8c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.886486 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886474 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjkqj\" (UniqueName: \"kubernetes.io/projected/d418480f-1d70-41ce-902f-c36c0c1b4b97-kube-api-access-tjkqj\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.886521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886493 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c37b6ad0-f8fd-4edd-81d9-0cab63ce6368-tmp\") pod \"downloads-6dd789c8b7-krv4w\" (UID: \"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368\") " pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:18.886521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886514 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p2w\" (UniqueName: \"kubernetes.io/projected/c37b6ad0-f8fd-4edd-81d9-0cab63ce6368-kube-api-access-62p2w\") pod \"downloads-6dd789c8b7-krv4w\" (UID: \"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368\") " pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:18.886583 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886532 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwnd\" (UniqueName: \"kubernetes.io/projected/da25fd18-f13d-419a-aae5-f9a74ba5d045-kube-api-access-clwnd\") pod \"ingress-canary-9slvd\" (UID: \"da25fd18-f13d-419a-aae5-f9a74ba5d045\") " pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:18.886583 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.886571 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d418480f-1d70-41ce-902f-c36c0c1b4b97-tmp-dir\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.892689 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.892669 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zt274\"" Oct 30 15:38:18.892689 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.892679 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 15:38:18.892993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.892975 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9c74v\"" Oct 30 15:38:18.897576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.897560 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 15:38:18.987344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987319 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf1a0534-442c-4393-9f6e-09811a28e8c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.987344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987347 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgdj\" (UniqueName: \"kubernetes.io/projected/cf1a0534-442c-4393-9f6e-09811a28e8c5-kube-api-access-zjgdj\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.987531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987363 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d418480f-1d70-41ce-902f-c36c0c1b4b97-config-volume\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.987531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987380 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da25fd18-f13d-419a-aae5-f9a74ba5d045-cert\") pod \"ingress-canary-9slvd\" (UID: \"da25fd18-f13d-419a-aae5-f9a74ba5d045\") " pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:18.987531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987399 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf1a0534-442c-4393-9f6e-09811a28e8c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.987531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987425 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjkqj\" (UniqueName: \"kubernetes.io/projected/d418480f-1d70-41ce-902f-c36c0c1b4b97-kube-api-access-tjkqj\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.987531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987449 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c37b6ad0-f8fd-4edd-81d9-0cab63ce6368-tmp\") pod \"downloads-6dd789c8b7-krv4w\" (UID: \"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368\") " pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:18.987531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987471 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p2w\" (UniqueName: \"kubernetes.io/projected/c37b6ad0-f8fd-4edd-81d9-0cab63ce6368-kube-api-access-62p2w\") pod \"downloads-6dd789c8b7-krv4w\" (UID: \"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368\") " pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:18.987531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987519 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clwnd\" (UniqueName: \"kubernetes.io/projected/da25fd18-f13d-419a-aae5-f9a74ba5d045-kube-api-access-clwnd\") pod \"ingress-canary-9slvd\" (UID: \"da25fd18-f13d-419a-aae5-f9a74ba5d045\") " pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:18.987905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987550 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d418480f-1d70-41ce-902f-c36c0c1b4b97-tmp-dir\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.987905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987588 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e61536a3-3592-485a-98bd-93cbae66c2cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rzswm\" (UID: \"e61536a3-3592-485a-98bd-93cbae66c2cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" Oct 30 15:38:18.987905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987619 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf1a0534-442c-4393-9f6e-09811a28e8c5-crio-socket\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.987905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987641 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d418480f-1d70-41ce-902f-c36c0c1b4b97-metrics-tls\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.987905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987674 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1a0534-442c-4393-9f6e-09811a28e8c5-data-volume\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.987905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.987898 4401 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 15:38:18.988196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.988017 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cf1a0534-442c-4393-9f6e-09811a28e8c5-crio-socket\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.988196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.988032 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf1a0534-442c-4393-9f6e-09811a28e8c5-data-volume\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.988196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.988130 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c37b6ad0-f8fd-4edd-81d9-0cab63ce6368-tmp\") pod \"downloads-6dd789c8b7-krv4w\" (UID: \"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368\") " pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:18.988196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.988149 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cf1a0534-442c-4393-9f6e-09811a28e8c5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.988385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.988252 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d418480f-1d70-41ce-902f-c36c0c1b4b97-config-volume\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.988446 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.988426 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d418480f-1d70-41ce-902f-c36c0c1b4b97-tmp-dir\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.990255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.990220 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d418480f-1d70-41ce-902f-c36c0c1b4b97-metrics-tls\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:18.990426 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.990404 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e61536a3-3592-485a-98bd-93cbae66c2cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rzswm\" (UID: \"e61536a3-3592-485a-98bd-93cbae66c2cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" Oct 30 15:38:18.990478 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.990416 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cf1a0534-442c-4393-9f6e-09811a28e8c5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:18.990543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:18.990527 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da25fd18-f13d-419a-aae5-f9a74ba5d045-cert\") pod \"ingress-canary-9slvd\" (UID: \"da25fd18-f13d-419a-aae5-f9a74ba5d045\") " pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:19.017034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.016974 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgdj\" (UniqueName: \"kubernetes.io/projected/cf1a0534-442c-4393-9f6e-09811a28e8c5-kube-api-access-zjgdj\") pod \"insights-runtime-extractor-nrn5l\" (UID: \"cf1a0534-442c-4393-9f6e-09811a28e8c5\") " pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:19.017136 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.017096 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p2w\" (UniqueName: \"kubernetes.io/projected/c37b6ad0-f8fd-4edd-81d9-0cab63ce6368-kube-api-access-62p2w\") pod \"downloads-6dd789c8b7-krv4w\" (UID: \"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368\") " pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:19.021474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.021454 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjkqj\" (UniqueName: \"kubernetes.io/projected/d418480f-1d70-41ce-902f-c36c0c1b4b97-kube-api-access-tjkqj\") pod \"dns-default-vd8pb\" (UID: \"d418480f-1d70-41ce-902f-c36c0c1b4b97\") " pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:19.026666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.026639 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwnd\" (UniqueName: \"kubernetes.io/projected/da25fd18-f13d-419a-aae5-f9a74ba5d045-kube-api-access-clwnd\") pod \"ingress-canary-9slvd\" (UID: \"da25fd18-f13d-419a-aae5-f9a74ba5d045\") " pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:19.063169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.063148 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrn5l" Oct 30 15:38:19.066528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.066512 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:19.071214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.071195 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" Oct 30 15:38:19.102117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.101618 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:19.112920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.111739 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9slvd" Oct 30 15:38:19.390113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.390086 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-krv4w"] Oct 30 15:38:19.393642 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:19.393559 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37b6ad0_f8fd_4edd_81d9_0cab63ce6368.slice/crio-f832be9cce125d0209ceb1347fbaf0e8cda9da142e52dfc5eab74b0403076d96 WatchSource:0}: Error finding container f832be9cce125d0209ceb1347fbaf0e8cda9da142e52dfc5eab74b0403076d96: Status 404 returned error can't find the container with id f832be9cce125d0209ceb1347fbaf0e8cda9da142e52dfc5eab74b0403076d96 Oct 30 15:38:19.393798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.393780 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrn5l"] Oct 30 15:38:19.396090 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.396070 4401 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:38:19.396379 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:19.396330 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1a0534_442c_4393_9f6e_09811a28e8c5.slice/crio-a95bc508ae22f752f2b1d1f4d7126d4d7aec89eba89637979c1f91aa0313ab31 WatchSource:0}: Error finding container a95bc508ae22f752f2b1d1f4d7126d4d7aec89eba89637979c1f91aa0313ab31: Status 404 returned error can't find the container with id a95bc508ae22f752f2b1d1f4d7126d4d7aec89eba89637979c1f91aa0313ab31 Oct 30 15:38:19.396463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.396445 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm"] Oct 30 15:38:19.398527 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:19.398500 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61536a3_3592_485a_98bd_93cbae66c2cf.slice/crio-aa0c2d450cfeab3f4948d55b907e2c83ad4da19719ac2818acfd04015066abe4 WatchSource:0}: Error finding container aa0c2d450cfeab3f4948d55b907e2c83ad4da19719ac2818acfd04015066abe4: Status 404 returned error can't find the container with id aa0c2d450cfeab3f4948d55b907e2c83ad4da19719ac2818acfd04015066abe4 Oct 30 15:38:19.466194 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.466167 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9slvd"] Oct 30 15:38:19.469993 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:19.469960 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda25fd18_f13d_419a_aae5_f9a74ba5d045.slice/crio-38d97b5f3ae26d1083413cc3e3da1a493c1ead5e569f18228ee1a7a881fa5d45 WatchSource:0}: Error finding container 38d97b5f3ae26d1083413cc3e3da1a493c1ead5e569f18228ee1a7a881fa5d45: Status 404 returned error can't find the container with id 38d97b5f3ae26d1083413cc3e3da1a493c1ead5e569f18228ee1a7a881fa5d45 Oct 30 15:38:19.477029 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.476899 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vd8pb"] Oct 30 15:38:19.479001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.478046 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" event={"ID":"e61536a3-3592-485a-98bd-93cbae66c2cf","Type":"ContainerStarted","Data":"aa0c2d450cfeab3f4948d55b907e2c83ad4da19719ac2818acfd04015066abe4"} Oct 30 15:38:19.479428 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.479337 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-krv4w" event={"ID":"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368","Type":"ContainerStarted","Data":"f832be9cce125d0209ceb1347fbaf0e8cda9da142e52dfc5eab74b0403076d96"} Oct 30 15:38:19.479702 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:19.479675 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd418480f_1d70_41ce_902f_c36c0c1b4b97.slice/crio-f6946b36e6984bb80a7bf5d814cbc0f038d6f22dec8a1b5e4cf8a0ba6ea1cfb0 WatchSource:0}: Error finding container f6946b36e6984bb80a7bf5d814cbc0f038d6f22dec8a1b5e4cf8a0ba6ea1cfb0: Status 404 returned error can't find the container with id f6946b36e6984bb80a7bf5d814cbc0f038d6f22dec8a1b5e4cf8a0ba6ea1cfb0 Oct 30 15:38:19.481322 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.481295 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrn5l" event={"ID":"cf1a0534-442c-4393-9f6e-09811a28e8c5","Type":"ContainerStarted","Data":"36452fd9e607439ae548b973348164c67fe69177fe77242d940518c23358f0d8"} Oct 30 15:38:19.481369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:19.481335 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrn5l" event={"ID":"cf1a0534-442c-4393-9f6e-09811a28e8c5","Type":"ContainerStarted","Data":"a95bc508ae22f752f2b1d1f4d7126d4d7aec89eba89637979c1f91aa0313ab31"} Oct 30 15:38:20.487226 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:20.487183 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9slvd" event={"ID":"da25fd18-f13d-419a-aae5-f9a74ba5d045","Type":"ContainerStarted","Data":"38d97b5f3ae26d1083413cc3e3da1a493c1ead5e569f18228ee1a7a881fa5d45"} Oct 30 15:38:20.490217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:20.490181 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrn5l" event={"ID":"cf1a0534-442c-4393-9f6e-09811a28e8c5","Type":"ContainerStarted","Data":"541b6dbadc9fcb1954bec9993a49dca5fc2be2015a8b3d152170ab5acecf9bdf"} Oct 30 15:38:20.491659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:20.491631 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vd8pb" event={"ID":"d418480f-1d70-41ce-902f-c36c0c1b4b97","Type":"ContainerStarted","Data":"f6946b36e6984bb80a7bf5d814cbc0f038d6f22dec8a1b5e4cf8a0ba6ea1cfb0"} Oct 30 15:38:21.498672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:21.498635 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" event={"ID":"e61536a3-3592-485a-98bd-93cbae66c2cf","Type":"ContainerStarted","Data":"7522f03f97ab30f2f99d7c6241afe9bdeba793b21651dd15babc00ea321de322"} Oct 30 15:38:21.546300 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:21.546183 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" podStartSLOduration=2.18361669 podStartE2EDuration="3.546166528s" totalImagesPullingTime="1.362549838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:21.545095612 +0000 UTC m=+10.244368772" watchObservedRunningTime="2026-10-30 15:38:21.546166528 +0000 UTC m=+10.245439686" Oct 30 15:38:21.794380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:21.794297 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cm4sh_31d9f9bb-0f4f-4110-9ad8-28ac0bbde949/dns-node-resolver/0.log" Oct 30 15:38:22.505643 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.505600 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrn5l" event={"ID":"cf1a0534-442c-4393-9f6e-09811a28e8c5","Type":"ContainerStarted","Data":"9b39ed1591018d6e956ce097f993228ee8cb3971883148ee3dd679ffd00075a0"} Oct 30 15:38:22.508807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.508337 4401 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 15:38:22.508807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.508368 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vd8pb" event={"ID":"d418480f-1d70-41ce-902f-c36c0c1b4b97","Type":"ContainerStarted","Data":"9427bcf9f9b5d838c005406a32af5b546051d98f70f0bcd6b9830677877e6794"} Oct 30 15:38:22.508807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.508408 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vd8pb" event={"ID":"d418480f-1d70-41ce-902f-c36c0c1b4b97","Type":"ContainerStarted","Data":"ee792d143ffc2a682487a65b485d4d97704e4fdd548e2dfbf7059c5f8f0c19a7"} Oct 30 15:38:22.508807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.508585 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:22.600905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.600858 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nrn5l" podStartSLOduration=2.177811244 podStartE2EDuration="4.600842174s" totalImagesPullingTime="2.42303093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:38:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:22.553953571 +0000 UTC m=+11.253226728" watchObservedRunningTime="2026-10-30 15:38:22.600842174 +0000 UTC m=+11.300115328" Oct 30 15:38:22.601096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.601037 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-vd8pb" podStartSLOduration=2.217415942 podStartE2EDuration="4.601029911s" totalImagesPullingTime="2.383613969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:22.600403194 +0000 UTC m=+11.299676351" watchObservedRunningTime="2026-10-30 15:38:22.601029911 +0000 UTC m=+11.300303067" Oct 30 15:38:22.790563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:22.790541 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c99x2_0addea81-c333-4eed-8afa-3762350172bc/node-ca/0.log" Oct 30 15:38:23.224574 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.224546 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-899bcffd6-qjngs"] Oct 30 15:38:23.226729 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.226703 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.248675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.248650 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2cbsx\"" Oct 30 15:38:23.260033 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.260008 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 15:38:23.262966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.262933 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-899bcffd6-qjngs"] Oct 30 15:38:23.263099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.263006 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 15:38:23.263099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.263080 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 15:38:23.263204 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.263110 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 15:38:23.312966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.312931 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-oauth-config\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.313158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.312978 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-tmp\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.313158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.313115 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-oauth-serving-cert\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.313158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.313153 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-serving-cert\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.313324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.313181 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-config\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.313324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.313207 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6hc\" (UniqueName: \"kubernetes.io/projected/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-kube-api-access-vs6hc\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.313324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.313251 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-service-ca\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.348242 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.348215 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 15:38:23.414349 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.414273 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-oauth-config\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.414349 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.414320 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-tmp\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.415137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.415076 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-tmp\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.415324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.415284 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-oauth-serving-cert\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.415897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.415519 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-serving-cert\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.415897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.415559 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-config\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.415897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.415584 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6hc\" (UniqueName: \"kubernetes.io/projected/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-kube-api-access-vs6hc\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.415897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.415633 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-service-ca\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.421847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.417994 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-oauth-config\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.421847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.420364 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-serving-cert\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.421847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.420502 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-oauth-serving-cert\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.421847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.420531 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-service-ca\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.425401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.425376 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-config\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.442711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.442690 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6hc\" (UniqueName: \"kubernetes.io/projected/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-kube-api-access-vs6hc\") pod \"console-899bcffd6-qjngs\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.527106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.526974 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9slvd" event={"ID":"da25fd18-f13d-419a-aae5-f9a74ba5d045","Type":"ContainerStarted","Data":"0027e8cf101d427e2515b82db92abffb392596711a218d00738136869c19945e"} Oct 30 15:38:23.537877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.537846 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:23.579018 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.578952 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9slvd" podStartSLOduration=2.310690803 podStartE2EDuration="5.578932272s" totalImagesPullingTime="3.268241469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:23.578274248 +0000 UTC m=+12.277547407" watchObservedRunningTime="2026-10-30 15:38:23.578932272 +0000 UTC m=+12.278205433" Oct 30 15:38:23.590856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.589976 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9slvd_da25fd18-f13d-419a-aae5-f9a74ba5d045/serve-healthcheck-canary/0.log" Oct 30 15:38:23.860125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:23.859147 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-899bcffd6-qjngs"] Oct 30 15:38:23.861983 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:23.861951 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a8c688_afc2_4e93_b6cc_56ef5d00e945.slice/crio-4a4865eec3555f490171ed06f2c2ea280921b5154ef4ea518e246c3d6d8302e5 WatchSource:0}: Error finding container 4a4865eec3555f490171ed06f2c2ea280921b5154ef4ea518e246c3d6d8302e5: Status 404 returned error can't find the container with id 4a4865eec3555f490171ed06f2c2ea280921b5154ef4ea518e246c3d6d8302e5 Oct 30 15:38:24.532272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:24.532234 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-899bcffd6-qjngs" event={"ID":"e3a8c688-afc2-4e93-b6cc-56ef5d00e945","Type":"ContainerStarted","Data":"4a4865eec3555f490171ed06f2c2ea280921b5154ef4ea518e246c3d6d8302e5"} Oct 30 15:38:27.258269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:27.258217 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" Oct 30 15:38:27.258820 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:27.258418 4401 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 15:38:27.262718 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:27.262695 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rzswm" Oct 30 15:38:28.548008 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:28.547974 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-899bcffd6-qjngs" event={"ID":"e3a8c688-afc2-4e93-b6cc-56ef5d00e945","Type":"ContainerStarted","Data":"00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be"} Oct 30 15:38:28.595549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:28.595491 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-899bcffd6-qjngs" podStartSLOduration=1.532115234 podStartE2EDuration="5.595477241s" totalImagesPullingTime="4.063362007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:28.594893371 +0000 UTC m=+17.294166530" watchObservedRunningTime="2026-10-30 15:38:28.595477241 +0000 UTC m=+17.294750389" Oct 30 15:38:32.529940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:32.529904 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vd8pb" Oct 30 15:38:32.975209 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:32.975172 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5"] Oct 30 15:38:32.978032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:32.978009 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.001251 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.001229 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 15:38:33.005662 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.005642 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 15:38:33.015785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.015737 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 15:38:33.015785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.015763 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 15:38:33.015977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.015918 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qjrf4\"" Oct 30 15:38:33.016898 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.016876 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5"] Oct 30 15:38:33.020402 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.020385 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85"] Oct 30 15:38:33.022905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.022875 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.049749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.049724 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85"] Oct 30 15:38:33.055602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.055583 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2wkrj"] Oct 30 15:38:33.057680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.057655 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.069220 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.069202 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 15:38:33.069613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.069590 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 15:38:33.097783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.097759 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.097921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.097802 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpgn\" (UniqueName: \"kubernetes.io/projected/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-kube-api-access-xkpgn\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.097921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.097887 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.098084 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.097927 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.101962 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.101944 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 15:38:33.107600 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.107580 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-8xsd2\"" Oct 30 15:38:33.107717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.107602 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 15:38:33.107717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.107605 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 15:38:33.107867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.107849 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 15:38:33.108405 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.108388 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 15:38:33.108461 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.108434 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bdgdg\"" Oct 30 15:38:33.159743 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.159719 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vkxh" Oct 30 15:38:33.198542 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198517 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpgn\" (UniqueName: \"kubernetes.io/projected/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-kube-api-access-xkpgn\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.198691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198550 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-textfile\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.198691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198579 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.198691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198596 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e03e2cfa-48dc-4593-9f45-5192472e0002-metrics-client-ca\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.198691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198660 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f324c092-4939-409c-9c05-b95a11343f8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.198918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198714 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.198918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198792 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sktz\" (UniqueName: \"kubernetes.io/projected/f324c092-4939-409c-9c05-b95a11343f8a-kube-api-access-9sktz\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.198918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198880 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f324c092-4939-409c-9c05-b95a11343f8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.198918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198907 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-root\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.199038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.198950 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.199038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199019 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.199120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199071 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-wtmp\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.199120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199106 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.199182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199144 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.199213 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199181 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.199244 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199207 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-sys\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.199244 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199232 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-tls\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.199304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199266 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wblbb\" (UniqueName: \"kubernetes.io/projected/e03e2cfa-48dc-4593-9f45-5192472e0002-kube-api-access-wblbb\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.199345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.199307 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.200103 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.200082 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.202794 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.202752 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.202794 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.202760 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.225504 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.225444 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpgn\" (UniqueName: \"kubernetes.io/projected/ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d-kube-api-access-xkpgn\") pod \"openshift-state-metrics-7bd7489957-qgtk5\" (UID: \"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.291326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.291287 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" Oct 30 15:38:33.300598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300570 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wblbb\" (UniqueName: \"kubernetes.io/projected/e03e2cfa-48dc-4593-9f45-5192472e0002-kube-api-access-wblbb\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.300692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300622 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-textfile\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.300692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300657 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.300692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300685 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e03e2cfa-48dc-4593-9f45-5192472e0002-metrics-client-ca\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.300876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300713 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f324c092-4939-409c-9c05-b95a11343f8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.300876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300762 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.300876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300814 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sktz\" (UniqueName: \"kubernetes.io/projected/f324c092-4939-409c-9c05-b95a11343f8a-kube-api-access-9sktz\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.300876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300860 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f324c092-4939-409c-9c05-b95a11343f8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300884 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-root\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300910 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300943 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.300968 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-wtmp\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.301026 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-textfile\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.301043 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.301059 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-sys\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.301214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.301080 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-tls\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.302097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.301889 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-wtmp\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.302097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.301945 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-root\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.302097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.301991 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e03e2cfa-48dc-4593-9f45-5192472e0002-sys\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.302319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.302296 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f324c092-4939-409c-9c05-b95a11343f8a-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.302370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.302324 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e03e2cfa-48dc-4593-9f45-5192472e0002-metrics-client-ca\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.302423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.302400 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f324c092-4939-409c-9c05-b95a11343f8a-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.302454 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.302428 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.303159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.303106 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-accelerators-collector-config\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.305266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.305237 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.305539 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.305516 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.305672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.305655 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f324c092-4939-409c-9c05-b95a11343f8a-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.305973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.305950 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e03e2cfa-48dc-4593-9f45-5192472e0002-node-exporter-tls\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.329281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.329227 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wblbb\" (UniqueName: \"kubernetes.io/projected/e03e2cfa-48dc-4593-9f45-5192472e0002-kube-api-access-wblbb\") pod \"node-exporter-2wkrj\" (UID: \"e03e2cfa-48dc-4593-9f45-5192472e0002\") " pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.347097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.347065 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sktz\" (UniqueName: \"kubernetes.io/projected/f324c092-4939-409c-9c05-b95a11343f8a-kube-api-access-9sktz\") pod \"kube-state-metrics-5dc8d7d89b-h5c85\" (UID: \"f324c092-4939-409c-9c05-b95a11343f8a\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.371342 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.371303 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2wkrj" Oct 30 15:38:33.382947 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:33.382889 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03e2cfa_48dc_4593_9f45_5192472e0002.slice/crio-375b7a5f6e6d1cf22b29eac6234a0556947112f6dff37f728fd466b258206e4c WatchSource:0}: Error finding container 375b7a5f6e6d1cf22b29eac6234a0556947112f6dff37f728fd466b258206e4c: Status 404 returned error can't find the container with id 375b7a5f6e6d1cf22b29eac6234a0556947112f6dff37f728fd466b258206e4c Oct 30 15:38:33.538521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.538448 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:33.538521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.538513 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:33.544967 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.544941 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:33.565867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.565818 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wkrj" event={"ID":"e03e2cfa-48dc-4593-9f45-5192472e0002","Type":"ContainerStarted","Data":"375b7a5f6e6d1cf22b29eac6234a0556947112f6dff37f728fd466b258206e4c"} Oct 30 15:38:33.570664 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.570639 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:38:33.612978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.612949 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5"] Oct 30 15:38:33.616220 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:33.616193 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb9f4fc_1e3c_482a_86fc_4a00dff61b5d.slice/crio-46afd33d2d37e79cfa4640fe02a0bc118255f6e5cbd922d8ed7670b47c1054be WatchSource:0}: Error finding container 46afd33d2d37e79cfa4640fe02a0bc118255f6e5cbd922d8ed7670b47c1054be: Status 404 returned error can't find the container with id 46afd33d2d37e79cfa4640fe02a0bc118255f6e5cbd922d8ed7670b47c1054be Oct 30 15:38:33.637707 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.637683 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" Oct 30 15:38:33.962246 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:33.962210 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85"] Oct 30 15:38:33.965612 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:33.965582 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf324c092_4939_409c_9c05_b95a11343f8a.slice/crio-44b3183c931c85a10b964f554986549517099ee686c95fbf9cede65a25cf78d0 WatchSource:0}: Error finding container 44b3183c931c85a10b964f554986549517099ee686c95fbf9cede65a25cf78d0: Status 404 returned error can't find the container with id 44b3183c931c85a10b964f554986549517099ee686c95fbf9cede65a25cf78d0 Oct 30 15:38:34.425773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.425725 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:38:34.432218 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.432188 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.467490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.467446 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 15:38:34.467654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.467550 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 15:38:34.467654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.467466 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 15:38:34.467780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.467677 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 15:38:34.469538 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.469520 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 15:38:34.493568 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.493539 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:38:34.553442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.553402 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 15:38:34.553961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.553489 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 15:38:34.553961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.553506 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 15:38:34.553961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.553783 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cbhg8\"" Oct 30 15:38:34.553961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.553932 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 15:38:34.572536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.572504 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" event={"ID":"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d","Type":"ContainerStarted","Data":"5c07d22598403840b2dcc5e1c01f5d6f9f1fe713e41b5cf86aa3787d6bff79d7"} Oct 30 15:38:34.572624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.572573 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" event={"ID":"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d","Type":"ContainerStarted","Data":"8cbca2ae0bfc7b16190d7598f2810a8a6c7a56e529173705d0099290c8bb9a3e"} Oct 30 15:38:34.572624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.572598 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" event={"ID":"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d","Type":"ContainerStarted","Data":"46afd33d2d37e79cfa4640fe02a0bc118255f6e5cbd922d8ed7670b47c1054be"} Oct 30 15:38:34.574930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.574768 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" event={"ID":"f324c092-4939-409c-9c05-b95a11343f8a","Type":"ContainerStarted","Data":"44b3183c931c85a10b964f554986549517099ee686c95fbf9cede65a25cf78d0"} Oct 30 15:38:34.615995 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.615969 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616009 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616028 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-web-config\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616045 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616061 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616075 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616094 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616114 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthm5\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-kube-api-access-lthm5\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616130 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616148 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616178 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616197 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-config-out\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.616301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.616210 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717466 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717394 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717466 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717434 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-config-out\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717466 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717461 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717518 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717573 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717609 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-web-config\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717635 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717661 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.717731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717683 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.718411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717734 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.718411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717782 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthm5\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-kube-api-access-lthm5\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.718411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717809 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.718411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.717863 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.719240 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.719211 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.724420 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.723469 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.724420 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.723587 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.724420 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.724379 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.725275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.725236 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.725606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.725549 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.726557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.726525 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.726657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.726521 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.727233 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.727202 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-config-volume\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.728332 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.728312 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-config-out\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.739452 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.739429 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.739641 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.739619 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-web-config\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.746346 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.746318 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthm5\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-kube-api-access-lthm5\") pod \"alertmanager-main-0\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:34.768004 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:34.767976 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:38:35.121266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.121229 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:38:35.124873 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:35.124812 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2180244_21b7_47e8_a03d_a03320295f6a.slice/crio-43fd0d0afca530375dbb42265ffaac0a78c395b833422f2754e4df245a05fcda WatchSource:0}: Error finding container 43fd0d0afca530375dbb42265ffaac0a78c395b833422f2754e4df245a05fcda: Status 404 returned error can't find the container with id 43fd0d0afca530375dbb42265ffaac0a78c395b833422f2754e4df245a05fcda Oct 30 15:38:35.220383 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.220351 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6d79665d9-l9kd2"] Oct 30 15:38:35.248652 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.248588 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.251385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.251296 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d79665d9-l9kd2"] Oct 30 15:38:35.325629 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.325596 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 15:38:35.325629 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.325622 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 15:38:35.325862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.325845 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8mgbp90q5iee6\"" Oct 30 15:38:35.325913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.325866 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 15:38:35.325913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.325894 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 15:38:35.326004 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.325950 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mkc2s\"" Oct 30 15:38:35.326004 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.325953 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 15:38:35.424108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424075 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-tls\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424128 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8934a-fc31-4810-9f1e-0e333fc95b02-metrics-client-ca\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424250 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrbnd\" (UniqueName: \"kubernetes.io/projected/b8b8934a-fc31-4810-9f1e-0e333fc95b02-kube-api-access-jrbnd\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424396 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424287 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424396 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424317 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-grpc-tls\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424505 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424463 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424556 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424500 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424556 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424524 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b8934a-fc31-4810-9f1e-0e333fc95b02-tmp\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.424654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.424552 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525632 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525563 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525632 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525616 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525641 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b8934a-fc31-4810-9f1e-0e333fc95b02-tmp\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525662 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525679 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-tls\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525705 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8934a-fc31-4810-9f1e-0e333fc95b02-metrics-client-ca\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525746 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrbnd\" (UniqueName: \"kubernetes.io/projected/b8b8934a-fc31-4810-9f1e-0e333fc95b02-kube-api-access-jrbnd\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525772 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.525854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.525800 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-grpc-tls\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.526198 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.526077 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8b8934a-fc31-4810-9f1e-0e333fc95b02-tmp\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.526726 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.526696 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8934a-fc31-4810-9f1e-0e333fc95b02-metrics-client-ca\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.529695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.529676 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-grpc-tls\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.529783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.529753 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.529843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.529778 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-tls\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.530506 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.530484 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.531227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.531205 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.534271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.534239 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b8b8934a-fc31-4810-9f1e-0e333fc95b02-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.553852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.553816 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrbnd\" (UniqueName: \"kubernetes.io/projected/b8b8934a-fc31-4810-9f1e-0e333fc95b02-kube-api-access-jrbnd\") pod \"thanos-querier-6d79665d9-l9kd2\" (UID: \"b8b8934a-fc31-4810-9f1e-0e333fc95b02\") " pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.582513 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.582482 4401 generic.go:356] "Generic (PLEG): container finished" podID="e03e2cfa-48dc-4593-9f45-5192472e0002" containerID="cb3c3003e2ba04636a336334a6707fdebcb370ef9bd87f662e51328ddd96b8a5" exitCode=0 Oct 30 15:38:35.582620 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.582575 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wkrj" event={"ID":"e03e2cfa-48dc-4593-9f45-5192472e0002","Type":"ContainerDied","Data":"cb3c3003e2ba04636a336334a6707fdebcb370ef9bd87f662e51328ddd96b8a5"} Oct 30 15:38:35.585111 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.585088 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" event={"ID":"ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d","Type":"ContainerStarted","Data":"77cde8e8cbd2461bc16f7a6bf0b9fc2e01762dd5d8825e5be18a8de6df2c45f5"} Oct 30 15:38:35.586327 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.586300 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerStarted","Data":"43fd0d0afca530375dbb42265ffaac0a78c395b833422f2754e4df245a05fcda"} Oct 30 15:38:35.660881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.660818 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:35.681208 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:35.681163 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-qgtk5" podStartSLOduration=2.499322862 podStartE2EDuration="3.681142781s" totalImagesPullingTime="1.181819919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:35.680370812 +0000 UTC m=+24.379643992" watchObservedRunningTime="2026-10-30 15:38:35.681142781 +0000 UTC m=+24.380415938" Oct 30 15:38:36.023506 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.023468 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6d79665d9-l9kd2"] Oct 30 15:38:36.030809 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:36.030777 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b8934a_fc31_4810_9f1e_0e333fc95b02.slice/crio-bd0cfa6576507a5b00a6d6f09cfef5680d0bb82bf7e13daaa101a6d1dcb074a9 WatchSource:0}: Error finding container bd0cfa6576507a5b00a6d6f09cfef5680d0bb82bf7e13daaa101a6d1dcb074a9: Status 404 returned error can't find the container with id bd0cfa6576507a5b00a6d6f09cfef5680d0bb82bf7e13daaa101a6d1dcb074a9 Oct 30 15:38:36.044264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.044233 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57557d995d-cvgsf"] Oct 30 15:38:36.062616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.062595 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.071355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.071331 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57557d995d-cvgsf"] Oct 30 15:38:36.084735 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.084694 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 15:38:36.231947 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.231895 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-oauth-serving-cert\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.232137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.231958 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9b0d30-057a-4579-8407-4ff8ad8742c5-tmp\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.232137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.231994 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-service-ca\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.232137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.232017 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-config\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.232137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.232071 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lgz\" (UniqueName: \"kubernetes.io/projected/5f9b0d30-057a-4579-8407-4ff8ad8742c5-kube-api-access-p2lgz\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.232137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.232092 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-serving-cert\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.232137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.232122 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-oauth-config\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.232137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.232141 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-trusted-ca-bundle\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.333303 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.332887 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-serving-cert\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.333657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.333575 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-oauth-config\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.333763 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.333697 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-trusted-ca-bundle\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.333763 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.333742 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-oauth-serving-cert\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.333900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.333777 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9b0d30-057a-4579-8407-4ff8ad8742c5-tmp\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.333900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.333819 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-service-ca\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.333900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.333888 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-config\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.334014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.333933 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lgz\" (UniqueName: \"kubernetes.io/projected/5f9b0d30-057a-4579-8407-4ff8ad8742c5-kube-api-access-p2lgz\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.334498 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.334472 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9b0d30-057a-4579-8407-4ff8ad8742c5-tmp\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.334639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.334614 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-oauth-serving-cert\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.335017 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.334999 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-service-ca\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.335389 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.335372 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-config\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.335506 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.335484 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-trusted-ca-bundle\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.336257 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.336084 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-serving-cert\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.336366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.336300 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-oauth-config\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.360262 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.360239 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lgz\" (UniqueName: \"kubernetes.io/projected/5f9b0d30-057a-4579-8407-4ff8ad8742c5-kube-api-access-p2lgz\") pod \"console-57557d995d-cvgsf\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.398098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.398067 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:36.592797 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.592716 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" event={"ID":"f324c092-4939-409c-9c05-b95a11343f8a","Type":"ContainerStarted","Data":"acb369ea67e7b149b3841588536a0e209d09fc7524abdd0412aa4b0da8bb5f45"} Oct 30 15:38:36.593406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.592851 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" event={"ID":"f324c092-4939-409c-9c05-b95a11343f8a","Type":"ContainerStarted","Data":"660b6fcf57e8a2d6794bbd44a09d1198e4b1eddc9d643db388d76b7bb48a9e39"} Oct 30 15:38:36.593406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.592883 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" event={"ID":"f324c092-4939-409c-9c05-b95a11343f8a","Type":"ContainerStarted","Data":"afcbf70725512707444a78b7b91dffe94aebfe6774bfd0b3b2050a18da781789"} Oct 30 15:38:36.595847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.595793 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wkrj" event={"ID":"e03e2cfa-48dc-4593-9f45-5192472e0002","Type":"ContainerStarted","Data":"d33e9910bec1fbef2003be0fec7c9685f0eec160bd82cb18d9962e06a79efa8e"} Oct 30 15:38:36.595990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.595902 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2wkrj" event={"ID":"e03e2cfa-48dc-4593-9f45-5192472e0002","Type":"ContainerStarted","Data":"bb2176e2c65d54eb481c696ef557e8c0998e72e73a15f2185bed7f93307d6267"} Oct 30 15:38:36.597349 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.597318 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" event={"ID":"b8b8934a-fc31-4810-9f1e-0e333fc95b02","Type":"ContainerStarted","Data":"bd0cfa6576507a5b00a6d6f09cfef5680d0bb82bf7e13daaa101a6d1dcb074a9"} Oct 30 15:38:36.642161 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.642122 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-h5c85" podStartSLOduration=2.998149852 podStartE2EDuration="4.642109337s" totalImagesPullingTime="1.643959485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:36.640916959 +0000 UTC m=+25.340190117" watchObservedRunningTime="2026-10-30 15:38:36.642109337 +0000 UTC m=+25.341382493" Oct 30 15:38:36.691666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.691618 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2wkrj" podStartSLOduration=2.437652957 podStartE2EDuration="3.691603573s" totalImagesPullingTime="1.174703045s" totalInitContainerRuntime="79.247571ms" isStatefulPod=true podCreationTimestamp="2026-10-30 15:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:36.691051689 +0000 UTC m=+25.390324845" watchObservedRunningTime="2026-10-30 15:38:36.691603573 +0000 UTC m=+25.390876729" Oct 30 15:38:36.734478 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:36.734451 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57557d995d-cvgsf"] Oct 30 15:38:36.738082 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:36.738050 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9b0d30_057a_4579_8407_4ff8ad8742c5.slice/crio-3bf1222cdcb321c1908c3e765fae972d7a4f39ca111bbeb4c6113e4c5f97b50d WatchSource:0}: Error finding container 3bf1222cdcb321c1908c3e765fae972d7a4f39ca111bbeb4c6113e4c5f97b50d: Status 404 returned error can't find the container with id 3bf1222cdcb321c1908c3e765fae972d7a4f39ca111bbeb4c6113e4c5f97b50d Oct 30 15:38:37.601959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.601926 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-krv4w" event={"ID":"c37b6ad0-f8fd-4edd-81d9-0cab63ce6368","Type":"ContainerStarted","Data":"7941e92e2024c6015302793097bec42fc85cc2ec41afc88a23fc443459f4dff5"} Oct 30 15:38:37.602387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.602087 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:37.603726 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.603700 4401 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-krv4w container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused" start-of-body= Oct 30 15:38:37.603872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.603748 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-krv4w" podUID="c37b6ad0-f8fd-4edd-81d9-0cab63ce6368" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.7:8080/\": dial tcp 10.133.0.7:8080: connect: connection refused" Oct 30 15:38:37.603872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.603757 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2180244-21b7-47e8-a03d-a03320295f6a" containerID="9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8" exitCode=0 Oct 30 15:38:37.603872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.603796 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8"} Oct 30 15:38:37.605520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.605495 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57557d995d-cvgsf" event={"ID":"5f9b0d30-057a-4579-8407-4ff8ad8742c5","Type":"ContainerStarted","Data":"6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02"} Oct 30 15:38:37.605638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.605532 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57557d995d-cvgsf" event={"ID":"5f9b0d30-057a-4579-8407-4ff8ad8742c5","Type":"ContainerStarted","Data":"3bf1222cdcb321c1908c3e765fae972d7a4f39ca111bbeb4c6113e4c5f97b50d"} Oct 30 15:38:37.649253 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.649159 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-krv4w" podStartSLOduration=1.665092939 podStartE2EDuration="19.64914636s" totalImagesPullingTime="17.984053421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:37.64858344 +0000 UTC m=+26.347856601" watchObservedRunningTime="2026-10-30 15:38:37.64914636 +0000 UTC m=+26.348419511" Oct 30 15:38:37.758441 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:37.758380 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57557d995d-cvgsf" podStartSLOduration=1.7583629200000002 podStartE2EDuration="1.75836292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:37.75780201 +0000 UTC m=+26.457075169" watchObservedRunningTime="2026-10-30 15:38:37.75836292 +0000 UTC m=+26.457636076" Oct 30 15:38:38.300143 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.300104 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw"] Oct 30 15:38:38.333875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.333806 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw"] Oct 30 15:38:38.335380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.333981 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.356537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.356307 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 15:38:38.356537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.356512 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 15:38:38.369966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.369942 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 15:38:38.370222 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.370171 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wxb5r\"" Oct 30 15:38:38.370330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.370231 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 15:38:38.370330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.370307 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 15:38:38.385896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.385871 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 15:38:38.457744 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.457696 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh8gg\" (UniqueName: \"kubernetes.io/projected/007dcfe4-0ea2-43db-b10a-66acd81d4e83-kube-api-access-dh8gg\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.457942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.457766 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.457942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.457803 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-federate-client-tls\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.457942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.457845 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-metrics-client-ca\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.457942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.457888 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-serving-certs-ca-bundle\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.458164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.458039 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-secret-telemeter-client\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.458164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.458070 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-telemeter-client-tls\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.458164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.458131 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.559769 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559687 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.559769 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559755 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh8gg\" (UniqueName: \"kubernetes.io/projected/007dcfe4-0ea2-43db-b10a-66acd81d4e83-kube-api-access-dh8gg\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559800 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559851 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-federate-client-tls\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559879 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-metrics-client-ca\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559927 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-serving-certs-ca-bundle\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559967 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-secret-telemeter-client\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.559993 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-telemeter-client-tls\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560726 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.560697 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-serving-certs-ca-bundle\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.560862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.560841 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.561320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.561296 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/007dcfe4-0ea2-43db-b10a-66acd81d4e83-metrics-client-ca\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.564719 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.564691 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.565150 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.565105 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-federate-client-tls\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.568510 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.568467 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-secret-telemeter-client\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.574266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.574230 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/007dcfe4-0ea2-43db-b10a-66acd81d4e83-telemeter-client-tls\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.587685 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.587498 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh8gg\" (UniqueName: \"kubernetes.io/projected/007dcfe4-0ea2-43db-b10a-66acd81d4e83-kube-api-access-dh8gg\") pod \"telemeter-client-7d5b554b4d-f5vcw\" (UID: \"007dcfe4-0ea2-43db-b10a-66acd81d4e83\") " pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:38.627834 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.627789 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-krv4w" Oct 30 15:38:38.712356 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:38.712316 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" Oct 30 15:38:39.099801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:39.099765 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw"] Oct 30 15:38:39.102234 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:39.102202 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007dcfe4_0ea2_43db_b10a_66acd81d4e83.slice/crio-2cb209f66c8b11a1e58881c515078dbb44e7d8a1bf73f59ddfd253db545b5967 WatchSource:0}: Error finding container 2cb209f66c8b11a1e58881c515078dbb44e7d8a1bf73f59ddfd253db545b5967: Status 404 returned error can't find the container with id 2cb209f66c8b11a1e58881c515078dbb44e7d8a1bf73f59ddfd253db545b5967 Oct 30 15:38:39.616006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:39.615973 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" event={"ID":"b8b8934a-fc31-4810-9f1e-0e333fc95b02","Type":"ContainerStarted","Data":"e35259366adb6a9e7e81c37d2da619becbdf445e34a576f1681812c5dc78ce98"} Oct 30 15:38:39.617261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:39.617235 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" event={"ID":"007dcfe4-0ea2-43db-b10a-66acd81d4e83","Type":"ContainerStarted","Data":"2cb209f66c8b11a1e58881c515078dbb44e7d8a1bf73f59ddfd253db545b5967"} Oct 30 15:38:40.632456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:40.632415 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" event={"ID":"b8b8934a-fc31-4810-9f1e-0e333fc95b02","Type":"ContainerStarted","Data":"031ea4af6de1b9216f2117da383acaae8e2c1c772158cfd9386b98041785edce"} Oct 30 15:38:40.632906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:40.632543 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" event={"ID":"b8b8934a-fc31-4810-9f1e-0e333fc95b02","Type":"ContainerStarted","Data":"856506de1e80d093dd17b6d0bbea5ad82e7b3afe5eda686897f1ef6a7bd83f04"} Oct 30 15:38:40.638292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:40.638252 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerStarted","Data":"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72"} Oct 30 15:38:40.638442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:40.638340 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerStarted","Data":"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35"} Oct 30 15:38:40.638442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:40.638383 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerStarted","Data":"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd"} Oct 30 15:38:40.638442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:40.638421 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerStarted","Data":"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884"} Oct 30 15:38:41.646558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:41.646519 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerStarted","Data":"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f"} Oct 30 15:38:42.657947 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.657902 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" event={"ID":"b8b8934a-fc31-4810-9f1e-0e333fc95b02","Type":"ContainerStarted","Data":"11983a4ba0b45d5cfabe08cd0ad8c7b50c20bf31aaad4c029130b27ced07ce57"} Oct 30 15:38:42.658385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.658020 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" event={"ID":"b8b8934a-fc31-4810-9f1e-0e333fc95b02","Type":"ContainerStarted","Data":"2a8f17e163f2f82833e8a3066d77c878ec2203f0fb639ab407b5128db68a344e"} Oct 30 15:38:42.658385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.658059 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" event={"ID":"b8b8934a-fc31-4810-9f1e-0e333fc95b02","Type":"ContainerStarted","Data":"18ef10742612c879775e1584704bc782ea6430c9307c15cc52a0e5e2a25f6c28"} Oct 30 15:38:42.658385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.658296 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:42.660360 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.660331 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" event={"ID":"007dcfe4-0ea2-43db-b10a-66acd81d4e83","Type":"ContainerStarted","Data":"f0ee2f30891e8ffc036419ece5d9e1e0aa2246dc5ecfbe90a515805856a11ff9"} Oct 30 15:38:42.664784 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.664755 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerStarted","Data":"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f"} Oct 30 15:38:42.717097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.716295 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" podStartSLOduration=2.566419287 podStartE2EDuration="7.716277009s" totalImagesPullingTime="5.149857722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:42.714632792 +0000 UTC m=+31.413905960" watchObservedRunningTime="2026-10-30 15:38:42.716277009 +0000 UTC m=+31.415550166" Oct 30 15:38:42.787727 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:42.775742 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.119819001 podStartE2EDuration="8.775723754s" totalImagesPullingTime="5.633292417s" totalInitContainerRuntime="22.612336ms" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:42.774247328 +0000 UTC m=+31.473520486" watchObservedRunningTime="2026-10-30 15:38:42.775723754 +0000 UTC m=+31.474996912" Oct 30 15:38:43.671923 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:43.671881 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" event={"ID":"007dcfe4-0ea2-43db-b10a-66acd81d4e83","Type":"ContainerStarted","Data":"cc404299e38668b5a246eb7ff9ebf925d86207fb4764366028aa199cd3a879ba"} Oct 30 15:38:43.672506 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:43.671951 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" event={"ID":"007dcfe4-0ea2-43db-b10a-66acd81d4e83","Type":"ContainerStarted","Data":"a7a970979c23610af98e56d56ca53add6b1f31e9941e6c5669074e61b9b46859"} Oct 30 15:38:43.761509 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:43.761449 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d5b554b4d-f5vcw" podStartSLOduration=2.420600656 podStartE2EDuration="5.761422059s" totalImagesPullingTime="3.340821403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:43.746862119 +0000 UTC m=+32.446135277" watchObservedRunningTime="2026-10-30 15:38:43.761422059 +0000 UTC m=+32.460695217" Oct 30 15:38:46.398665 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:46.398634 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:46.399059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:46.398936 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:46.403433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:46.403410 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:46.686174 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:46.686099 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:38:46.785230 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:46.785201 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-899bcffd6-qjngs"] Oct 30 15:38:47.267043 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.267016 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5975b97954-2hglk"] Oct 30 15:38:47.294933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.294907 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.299399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.299371 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5975b97954-2hglk"] Oct 30 15:38:47.348587 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348563 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faf41440-3225-4b97-ba2d-862c7cd5628b-tmp\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.348698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348593 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-serving-cert\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.348698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348622 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-oauth-config\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.348777 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348707 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-service-ca\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.348777 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348740 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-oauth-serving-cert\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.348777 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348763 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-console-config\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.348922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348883 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wfb\" (UniqueName: \"kubernetes.io/projected/faf41440-3225-4b97-ba2d-862c7cd5628b-kube-api-access-z5wfb\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.348973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.348922 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-trusted-ca-bundle\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.449732 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449702 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faf41440-3225-4b97-ba2d-862c7cd5628b-tmp\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449736 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-serving-cert\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449774 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-oauth-config\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449796 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-service-ca\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449819 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-oauth-serving-cert\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449866 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-console-config\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449926 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wfb\" (UniqueName: \"kubernetes.io/projected/faf41440-3225-4b97-ba2d-862c7cd5628b-kube-api-access-z5wfb\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.449955 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-trusted-ca-bundle\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.450163 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faf41440-3225-4b97-ba2d-862c7cd5628b-tmp\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.450718 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-service-ca\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.450726 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-console-config\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.450788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.450732 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-oauth-serving-cert\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.451046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.451028 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-trusted-ca-bundle\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.452295 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.452272 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-serving-cert\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.452430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.452412 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-oauth-config\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.477228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.477200 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wfb\" (UniqueName: \"kubernetes.io/projected/faf41440-3225-4b97-ba2d-862c7cd5628b-kube-api-access-z5wfb\") pod \"console-5975b97954-2hglk\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.620015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.619990 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:47.946144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:47.946120 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5975b97954-2hglk"] Oct 30 15:38:47.947848 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:38:47.947802 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf41440_3225_4b97_ba2d_862c7cd5628b.slice/crio-1f8d3847ee24959ddc4787fb0e902093f6c70560c7d87ca2680d463d96ea91fc WatchSource:0}: Error finding container 1f8d3847ee24959ddc4787fb0e902093f6c70560c7d87ca2680d463d96ea91fc: Status 404 returned error can't find the container with id 1f8d3847ee24959ddc4787fb0e902093f6c70560c7d87ca2680d463d96ea91fc Oct 30 15:38:48.513660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:48.513632 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tcrzp" Oct 30 15:38:48.677557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:48.677532 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6d79665d9-l9kd2" Oct 30 15:38:48.691898 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:48.691864 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5975b97954-2hglk" event={"ID":"faf41440-3225-4b97-ba2d-862c7cd5628b","Type":"ContainerStarted","Data":"ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7"} Oct 30 15:38:48.692024 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:48.691910 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5975b97954-2hglk" event={"ID":"faf41440-3225-4b97-ba2d-862c7cd5628b","Type":"ContainerStarted","Data":"1f8d3847ee24959ddc4787fb0e902093f6c70560c7d87ca2680d463d96ea91fc"} Oct 30 15:38:48.777968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:48.777878 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5975b97954-2hglk" podStartSLOduration=1.777860171 podStartE2EDuration="1.777860171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:38:48.776159874 +0000 UTC m=+37.475433031" watchObservedRunningTime="2026-10-30 15:38:48.777860171 +0000 UTC m=+37.477133329" Oct 30 15:38:51.964565 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:38:51.964528 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:38:57.621098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:57.621024 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:57.621098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:57.621087 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:57.625500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:57.625479 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:57.729788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:57.729758 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:38:57.811143 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:38:57.811114 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57557d995d-cvgsf"] Oct 30 15:39:11.810566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:11.810506 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-899bcffd6-qjngs" podUID="e3a8c688-afc2-4e93-b6cc-56ef5d00e945" containerName="console" containerID="cri-o://00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be" gracePeriod=15 Oct 30 15:39:12.096985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.096956 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:39:12.262077 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262042 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-kube-api-access-vs6hc\" (UniqueName: \"kubernetes.io/projected/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-kube-api-access-vs6hc\") pod \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " Oct 30 15:39:12.262261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262097 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-service-ca\") pod \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " Oct 30 15:39:12.262261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262126 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-config\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-config\") pod \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " Oct 30 15:39:12.262261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262156 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-oauth-config\") pod \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " Oct 30 15:39:12.262261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262199 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-tmp\") pod \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " Oct 30 15:39:12.262261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262237 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-oauth-serving-cert\") pod \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " Oct 30 15:39:12.262518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262301 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-serving-cert\") pod \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\" (UID: \"e3a8c688-afc2-4e93-b6cc-56ef5d00e945\") " Oct 30 15:39:12.262596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262548 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-tmp" pod "e3a8c688-afc2-4e93-b6cc-56ef5d00e945" (UID: "e3a8c688-afc2-4e93-b6cc-56ef5d00e945"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:12.262672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262601 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-oauth-serving-cert" pod "e3a8c688-afc2-4e93-b6cc-56ef5d00e945" (UID: "e3a8c688-afc2-4e93-b6cc-56ef5d00e945"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:12.262672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262568 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-config" pod "e3a8c688-afc2-4e93-b6cc-56ef5d00e945" (UID: "e3a8c688-afc2-4e93-b6cc-56ef5d00e945"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:12.262672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.262577 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-service-ca" pod "e3a8c688-afc2-4e93-b6cc-56ef5d00e945" (UID: "e3a8c688-afc2-4e93-b6cc-56ef5d00e945"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:12.264446 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.264424 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-oauth-config" pod "e3a8c688-afc2-4e93-b6cc-56ef5d00e945" (UID: "e3a8c688-afc2-4e93-b6cc-56ef5d00e945"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:12.264538 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.264446 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-kube-api-access-vs6hc" pod "e3a8c688-afc2-4e93-b6cc-56ef5d00e945" (UID: "e3a8c688-afc2-4e93-b6cc-56ef5d00e945"). InnerVolumeSpecName "kube-api-access-vs6hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:12.264538 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.264469 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-serving-cert" pod "e3a8c688-afc2-4e93-b6cc-56ef5d00e945" (UID: "e3a8c688-afc2-4e93-b6cc-56ef5d00e945"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:12.363819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.363798 4401 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-serving-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:12.363944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.363820 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6hc\" (UniqueName: \"kubernetes.io/projected/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-kube-api-access-vs6hc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:12.363944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.363850 4401 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:12.363944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.363864 4401 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:12.363944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.363876 4401 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-console-oauth-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:12.363944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.363884 4401 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-tmp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:12.363944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.363893 4401 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e3a8c688-afc2-4e93-b6cc-56ef5d00e945-oauth-serving-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:12.779410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.779324 4401 generic.go:356] "Generic (PLEG): container finished" podID="e3a8c688-afc2-4e93-b6cc-56ef5d00e945" containerID="00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be" exitCode=0 Oct 30 15:39:12.779410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.779368 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-899bcffd6-qjngs" event={"ID":"e3a8c688-afc2-4e93-b6cc-56ef5d00e945","Type":"ContainerDied","Data":"00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be"} Oct 30 15:39:12.779410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.779383 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-899bcffd6-qjngs" Oct 30 15:39:12.779410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.779402 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-899bcffd6-qjngs" event={"ID":"e3a8c688-afc2-4e93-b6cc-56ef5d00e945","Type":"ContainerDied","Data":"4a4865eec3555f490171ed06f2c2ea280921b5154ef4ea518e246c3d6d8302e5"} Oct 30 15:39:12.779717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.779420 4401 scope.go:122] "RemoveContainer" containerID="00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be" Oct 30 15:39:12.788210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.788193 4401 scope.go:122] "RemoveContainer" containerID="00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be" Oct 30 15:39:12.788503 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:12.788485 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be\": container with ID starting with 00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be not found: ID does not exist" containerID="00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be" Oct 30 15:39:12.788553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.788514 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be"} err="failed to get container status \"00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be\": rpc error: code = NotFound desc = could not find container \"00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be\": container with ID starting with 00878001277e79bf912efb01482b002cf3eea1331ab0f80639de8d909dedc5be not found: ID does not exist" Oct 30 15:39:12.836095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.836074 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-899bcffd6-qjngs"] Oct 30 15:39:12.850722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:12.850694 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-899bcffd6-qjngs"] Oct 30 15:39:13.424836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:13.424792 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a8c688-afc2-4e93-b6cc-56ef5d00e945" path="/var/lib/kubelet/pods/e3a8c688-afc2-4e93-b6cc-56ef5d00e945/volumes" Oct 30 15:39:21.964582 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:21.964547 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:39:22.831369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:22.831334 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-57557d995d-cvgsf" podUID="5f9b0d30-057a-4579-8407-4ff8ad8742c5" containerName="console" containerID="cri-o://6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02" gracePeriod=15 Oct 30 15:39:23.093843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.093777 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:39:23.248208 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248180 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-config\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-config\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248214 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f9b0d30-057a-4579-8407-4ff8ad8742c5-kube-api-access-p2lgz\" (UniqueName: \"kubernetes.io/projected/5f9b0d30-057a-4579-8407-4ff8ad8742c5-kube-api-access-p2lgz\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248243 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-service-ca\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248264 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f9b0d30-057a-4579-8407-4ff8ad8742c5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9b0d30-057a-4579-8407-4ff8ad8742c5-tmp\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248303 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-trusted-ca-bundle\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248323 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-serving-cert\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248370 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-oauth-serving-cert\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248411 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-oauth-config\") pod \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\" (UID: \"5f9b0d30-057a-4579-8407-4ff8ad8742c5\") " Oct 30 15:39:23.248634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248585 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9b0d30-057a-4579-8407-4ff8ad8742c5-tmp" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:23.248779 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248692 4401 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f9b0d30-057a-4579-8407-4ff8ad8742c5-tmp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.248779 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248715 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-config" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:23.248779 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248726 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-service-ca" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:23.249006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.248754 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-trusted-ca-bundle" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:23.250395 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.249075 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-oauth-serving-cert" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:23.251102 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.251026 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9b0d30-057a-4579-8407-4ff8ad8742c5-kube-api-access-p2lgz" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "kube-api-access-p2lgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:23.251507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.251478 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-oauth-config" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:23.254806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.254773 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-serving-cert" pod "5f9b0d30-057a-4579-8407-4ff8ad8742c5" (UID: "5f9b0d30-057a-4579-8407-4ff8ad8742c5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:23.301600 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.301576 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b44548658-8dpd8"] Oct 30 15:39:23.301941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.301928 4401 cpu_manager.go:414] "containerMap: removing container" podUID="5f9b0d30-057a-4579-8407-4ff8ad8742c5" containerName="console" Oct 30 15:39:23.301989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.301943 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f9b0d30-057a-4579-8407-4ff8ad8742c5" containerName="console" Oct 30 15:39:23.301989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.301957 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e3a8c688-afc2-4e93-b6cc-56ef5d00e945" containerName="console" Oct 30 15:39:23.301989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.301962 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a8c688-afc2-4e93-b6cc-56ef5d00e945" containerName="console" Oct 30 15:39:23.302080 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.302003 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f9b0d30-057a-4579-8407-4ff8ad8742c5" containerName="console" Oct 30 15:39:23.302080 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.302012 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a8c688-afc2-4e93-b6cc-56ef5d00e945" containerName="console" Oct 30 15:39:23.320968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.320947 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.324536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.324513 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b44548658-8dpd8"] Oct 30 15:39:23.349525 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.349505 4401 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.349619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.349525 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lgz\" (UniqueName: \"kubernetes.io/projected/5f9b0d30-057a-4579-8407-4ff8ad8742c5-kube-api-access-p2lgz\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.349619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.349536 4401 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.349619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.349545 4401 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-trusted-ca-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.349619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.349554 4401 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-serving-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.349619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.349563 4401 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5f9b0d30-057a-4579-8407-4ff8ad8742c5-oauth-serving-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.349619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.349571 4401 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5f9b0d30-057a-4579-8407-4ff8ad8742c5-console-oauth-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:23.450889 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.450864 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a93c916-efc9-4792-8e71-36faa1839652-console-oauth-config\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.450989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.450893 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m7nc\" (UniqueName: \"kubernetes.io/projected/0a93c916-efc9-4792-8e71-36faa1839652-kube-api-access-5m7nc\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.450989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.450922 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-trusted-ca-bundle\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.451064 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.451054 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-console-config\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.451097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.451077 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a93c916-efc9-4792-8e71-36faa1839652-tmp\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.451128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.451116 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a93c916-efc9-4792-8e71-36faa1839652-console-serving-cert\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.451241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.451224 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-service-ca\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.451272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.451249 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-oauth-serving-cert\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552072 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-trusted-ca-bundle\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552115 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-console-config\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552132 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a93c916-efc9-4792-8e71-36faa1839652-tmp\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552169 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a93c916-efc9-4792-8e71-36faa1839652-console-serving-cert\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552189 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-service-ca\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552241 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-oauth-serving-cert\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552270 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a93c916-efc9-4792-8e71-36faa1839652-console-oauth-config\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552296 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m7nc\" (UniqueName: \"kubernetes.io/projected/0a93c916-efc9-4792-8e71-36faa1839652-kube-api-access-5m7nc\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.552903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552877 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-console-config\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.553018 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.552951 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-service-ca\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.553084 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.553016 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a93c916-efc9-4792-8e71-36faa1839652-tmp\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.553084 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.553041 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-oauth-serving-cert\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.553196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.553145 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a93c916-efc9-4792-8e71-36faa1839652-trusted-ca-bundle\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.554592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.554565 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0a93c916-efc9-4792-8e71-36faa1839652-console-oauth-config\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.554686 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.554670 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0a93c916-efc9-4792-8e71-36faa1839652-console-serving-cert\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.577202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.577184 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m7nc\" (UniqueName: \"kubernetes.io/projected/0a93c916-efc9-4792-8e71-36faa1839652-kube-api-access-5m7nc\") pod \"console-5b44548658-8dpd8\" (UID: \"0a93c916-efc9-4792-8e71-36faa1839652\") " pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.643846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.643777 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:23.819050 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.819015 4401 generic.go:356] "Generic (PLEG): container finished" podID="5f9b0d30-057a-4579-8407-4ff8ad8742c5" containerID="6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02" exitCode=0 Oct 30 15:39:23.819239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.819073 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57557d995d-cvgsf" event={"ID":"5f9b0d30-057a-4579-8407-4ff8ad8742c5","Type":"ContainerDied","Data":"6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02"} Oct 30 15:39:23.819239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.819093 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57557d995d-cvgsf" Oct 30 15:39:23.819239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.819115 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57557d995d-cvgsf" event={"ID":"5f9b0d30-057a-4579-8407-4ff8ad8742c5","Type":"ContainerDied","Data":"3bf1222cdcb321c1908c3e765fae972d7a4f39ca111bbeb4c6113e4c5f97b50d"} Oct 30 15:39:23.819239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.819135 4401 scope.go:122] "RemoveContainer" containerID="6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02" Oct 30 15:39:23.826820 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.826798 4401 scope.go:122] "RemoveContainer" containerID="6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02" Oct 30 15:39:23.827085 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:23.827059 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02\": container with ID starting with 6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02 not found: ID does not exist" containerID="6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02" Oct 30 15:39:23.827131 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.827087 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02"} err="failed to get container status \"6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02\": rpc error: code = NotFound desc = could not find container \"6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02\": container with ID starting with 6c4ce458047df0fd254da81db8f02eb11dd2ae4314550bc46252bf96121dab02 not found: ID does not exist" Oct 30 15:39:23.875110 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.875092 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57557d995d-cvgsf"] Oct 30 15:39:23.888943 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.888910 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57557d995d-cvgsf"] Oct 30 15:39:23.946986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.946937 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:39:23.947255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.947211 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="alertmanager" containerID="cri-o://690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884" gracePeriod=120 Oct 30 15:39:23.947315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.947258 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-metric" containerID="cri-o://596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f" gracePeriod=120 Oct 30 15:39:23.947315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.947288 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-web" containerID="cri-o://d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35" gracePeriod=120 Oct 30 15:39:23.947419 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.947320 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="prom-label-proxy" containerID="cri-o://c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f" gracePeriod=120 Oct 30 15:39:23.947419 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.947356 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy" containerID="cri-o://2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72" gracePeriod=120 Oct 30 15:39:23.947419 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.947307 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="config-reloader" containerID="cri-o://3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd" gracePeriod=120 Oct 30 15:39:23.970061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:23.970042 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b44548658-8dpd8"] Oct 30 15:39:24.825697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825665 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2180244-21b7-47e8-a03d-a03320295f6a" containerID="c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f" exitCode=0 Oct 30 15:39:24.825697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825688 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2180244-21b7-47e8-a03d-a03320295f6a" containerID="2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72" exitCode=0 Oct 30 15:39:24.825697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825696 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2180244-21b7-47e8-a03d-a03320295f6a" containerID="3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd" exitCode=0 Oct 30 15:39:24.825697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825703 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2180244-21b7-47e8-a03d-a03320295f6a" containerID="690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884" exitCode=0 Oct 30 15:39:24.826358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825732 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f"} Oct 30 15:39:24.826358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825791 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72"} Oct 30 15:39:24.826358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825837 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd"} Oct 30 15:39:24.826358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.825870 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884"} Oct 30 15:39:24.828033 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.828011 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b44548658-8dpd8" event={"ID":"0a93c916-efc9-4792-8e71-36faa1839652","Type":"ContainerStarted","Data":"595b31b1ffbe4c1db0d1ae4a4e953a3fcaf3a8ea16a14ab14876351214884448"} Oct 30 15:39:24.828125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.828046 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b44548658-8dpd8" event={"ID":"0a93c916-efc9-4792-8e71-36faa1839652","Type":"ContainerStarted","Data":"56da8f699ec9e27e76410866dd6a1b865049d153fd851c81f79e155798386a25"} Oct 30 15:39:24.875622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:24.875580 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b44548658-8dpd8" podStartSLOduration=1.8755693 podStartE2EDuration="1.8755693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:24.87420482 +0000 UTC m=+73.573477976" watchObservedRunningTime="2026-10-30 15:39:24.8755693 +0000 UTC m=+73.574842457" Oct 30 15:39:25.220278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.220251 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:25.272044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272017 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-kube-api-access-lthm5\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-kube-api-access-lthm5\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272197 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272055 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-web-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-web-config\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272197 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272107 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-main-tls\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272197 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272138 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-config-out\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272197 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272172 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-tls-assets\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-tls-assets\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272199 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-config-volume\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-config-volume\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272246 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-trusted-ca-bundle\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272293 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-cluster-tls-config\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272322 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-web\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272357 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-main-db\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272404 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272434 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.272644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.272471 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-metrics-client-ca\") pod \"a2180244-21b7-47e8-a03d-a03320295f6a\" (UID: \"a2180244-21b7-47e8-a03d-a03320295f6a\") " Oct 30 15:39:25.273108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.273079 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-metrics-client-ca" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:25.273314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.273144 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-main-db" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:25.273416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.273366 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-trusted-ca-bundle" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:25.275048 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.275012 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-kube-api-access-lthm5" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "kube-api-access-lthm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:25.276278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.276191 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-config-volume" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:25.276278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.276204 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-config-out" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:25.276433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.276276 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-tls-assets" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:25.276433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.276274 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-web" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:25.276433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.276381 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-main-tls" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:25.276563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.276486 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-metric" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:25.276596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.276570 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:25.279611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.279518 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-cluster-tls-config" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:25.285991 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.285968 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-web-config" pod "a2180244-21b7-47e8-a03d-a03320295f6a" (UID: "a2180244-21b7-47e8-a03d-a03320295f6a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:25.373758 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373727 4401 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373758 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373754 4401 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-cluster-tls-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373764 4401 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373773 4401 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-alertmanager-main-db\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373783 4401 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373791 4401 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373800 4401 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2180244-21b7-47e8-a03d-a03320295f6a-metrics-client-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373808 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lthm5\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-kube-api-access-lthm5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373817 4401 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-web-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373835 4401 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-secret-alertmanager-main-tls\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373843 4401 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a2180244-21b7-47e8-a03d-a03320295f6a-config-out\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373851 4401 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a2180244-21b7-47e8-a03d-a03320295f6a-tls-assets\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.373959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.373858 4401 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a2180244-21b7-47e8-a03d-a03320295f6a-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.429013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.426907 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f9b0d30-057a-4579-8407-4ff8ad8742c5" path="/var/lib/kubelet/pods/5f9b0d30-057a-4579-8407-4ff8ad8742c5/volumes" Oct 30 15:39:25.834710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.834628 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2180244-21b7-47e8-a03d-a03320295f6a" containerID="596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f" exitCode=0 Oct 30 15:39:25.834710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.834658 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2180244-21b7-47e8-a03d-a03320295f6a" containerID="d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35" exitCode=0 Oct 30 15:39:25.835153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.834714 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f"} Oct 30 15:39:25.835153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.834735 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:25.835153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.834795 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35"} Oct 30 15:39:25.835153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.834839 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a2180244-21b7-47e8-a03d-a03320295f6a","Type":"ContainerDied","Data":"43fd0d0afca530375dbb42265ffaac0a78c395b833422f2754e4df245a05fcda"} Oct 30 15:39:25.835153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.834848 4401 scope.go:122] "RemoveContainer" containerID="c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f" Oct 30 15:39:25.842167 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.842137 4401 scope.go:122] "RemoveContainer" containerID="596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f" Oct 30 15:39:25.849055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.849042 4401 scope.go:122] "RemoveContainer" containerID="2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72" Oct 30 15:39:25.855425 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.855409 4401 scope.go:122] "RemoveContainer" containerID="d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35" Oct 30 15:39:25.861666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.861649 4401 scope.go:122] "RemoveContainer" containerID="3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd" Oct 30 15:39:25.868694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.868678 4401 scope.go:122] "RemoveContainer" containerID="690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884" Oct 30 15:39:25.875476 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.875461 4401 scope.go:122] "RemoveContainer" containerID="9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8" Oct 30 15:39:25.881970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.881954 4401 scope.go:122] "RemoveContainer" containerID="c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f" Oct 30 15:39:25.882222 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:25.882202 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f\": container with ID starting with c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f not found: ID does not exist" containerID="c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f" Oct 30 15:39:25.882264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.882229 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f"} err="failed to get container status \"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f\": rpc error: code = NotFound desc = could not find container \"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f\": container with ID starting with c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f not found: ID does not exist" Oct 30 15:39:25.882264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.882246 4401 scope.go:122] "RemoveContainer" containerID="596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f" Oct 30 15:39:25.882484 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:25.882469 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f\": container with ID starting with 596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f not found: ID does not exist" containerID="596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f" Oct 30 15:39:25.882543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.882487 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f"} err="failed to get container status \"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f\": rpc error: code = NotFound desc = could not find container \"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f\": container with ID starting with 596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f not found: ID does not exist" Oct 30 15:39:25.882543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.882499 4401 scope.go:122] "RemoveContainer" containerID="2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72" Oct 30 15:39:25.882686 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:25.882673 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72\": container with ID starting with 2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72 not found: ID does not exist" containerID="2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72" Oct 30 15:39:25.882723 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.882688 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72"} err="failed to get container status \"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72\": rpc error: code = NotFound desc = could not find container \"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72\": container with ID starting with 2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72 not found: ID does not exist" Oct 30 15:39:25.882723 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.882698 4401 scope.go:122] "RemoveContainer" containerID="d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35" Oct 30 15:39:25.883166 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:25.883150 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35\": container with ID starting with d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35 not found: ID does not exist" containerID="d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35" Oct 30 15:39:25.883231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883165 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35"} err="failed to get container status \"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35\": rpc error: code = NotFound desc = could not find container \"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35\": container with ID starting with d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35 not found: ID does not exist" Oct 30 15:39:25.883231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883176 4401 scope.go:122] "RemoveContainer" containerID="3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd" Oct 30 15:39:25.883394 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:25.883378 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd\": container with ID starting with 3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd not found: ID does not exist" containerID="3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd" Oct 30 15:39:25.883432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883399 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd"} err="failed to get container status \"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd\": rpc error: code = NotFound desc = could not find container \"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd\": container with ID starting with 3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd not found: ID does not exist" Oct 30 15:39:25.883432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883414 4401 scope.go:122] "RemoveContainer" containerID="690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884" Oct 30 15:39:25.883614 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:25.883600 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884\": container with ID starting with 690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884 not found: ID does not exist" containerID="690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884" Oct 30 15:39:25.883666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883618 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884"} err="failed to get container status \"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884\": rpc error: code = NotFound desc = could not find container \"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884\": container with ID starting with 690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884 not found: ID does not exist" Oct 30 15:39:25.883666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883629 4401 scope.go:122] "RemoveContainer" containerID="9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8" Oct 30 15:39:25.883852 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:25.883837 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8\": container with ID starting with 9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8 not found: ID does not exist" containerID="9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8" Oct 30 15:39:25.883905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883855 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8"} err="failed to get container status \"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8\": rpc error: code = NotFound desc = could not find container \"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8\": container with ID starting with 9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8 not found: ID does not exist" Oct 30 15:39:25.883905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.883866 4401 scope.go:122] "RemoveContainer" containerID="c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f" Oct 30 15:39:25.884064 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884047 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f"} err="failed to get container status \"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f\": rpc error: code = NotFound desc = could not find container \"c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f\": container with ID starting with c975d65cd675301c1a1893077d7847220b6f975cff3230ab7076f5d5c854368f not found: ID does not exist" Oct 30 15:39:25.884113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884064 4401 scope.go:122] "RemoveContainer" containerID="596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f" Oct 30 15:39:25.884319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884297 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f"} err="failed to get container status \"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f\": rpc error: code = NotFound desc = could not find container \"596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f\": container with ID starting with 596083b7cc0a216db79552068a1c26212ba8e6355b5cf6255da79e99ea62752f not found: ID does not exist" Oct 30 15:39:25.884383 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884319 4401 scope.go:122] "RemoveContainer" containerID="2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72" Oct 30 15:39:25.884499 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884481 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72"} err="failed to get container status \"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72\": rpc error: code = NotFound desc = could not find container \"2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72\": container with ID starting with 2c0ab81ff36ec989b4c71bc33e5f71dbec56664c82956f804a28efec4c52fe72 not found: ID does not exist" Oct 30 15:39:25.884550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884499 4401 scope.go:122] "RemoveContainer" containerID="d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35" Oct 30 15:39:25.884740 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884718 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35"} err="failed to get container status \"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35\": rpc error: code = NotFound desc = could not find container \"d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35\": container with ID starting with d5867eabb846f4153a17d81ba7afa6400ae203cdd811e4858f61528531f2aa35 not found: ID does not exist" Oct 30 15:39:25.884802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884742 4401 scope.go:122] "RemoveContainer" containerID="3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd" Oct 30 15:39:25.885011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.884994 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd"} err="failed to get container status \"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd\": rpc error: code = NotFound desc = could not find container \"3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd\": container with ID starting with 3a877657b8a6a34f2d1ebde6247b603e1023ed58cbe5341d83bad171d8dbb5dd not found: ID does not exist" Oct 30 15:39:25.885060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.885012 4401 scope.go:122] "RemoveContainer" containerID="690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884" Oct 30 15:39:25.885253 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.885230 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884"} err="failed to get container status \"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884\": rpc error: code = NotFound desc = could not find container \"690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884\": container with ID starting with 690bf811678e8a6b3025f75883b70c53768c1494d461f8da44263be0cc680884 not found: ID does not exist" Oct 30 15:39:25.885333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.885254 4401 scope.go:122] "RemoveContainer" containerID="9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8" Oct 30 15:39:25.885476 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.885452 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8"} err="failed to get container status \"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8\": rpc error: code = NotFound desc = could not find container \"9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8\": container with ID starting with 9e37f01ed539fe74419623729a8643edd6ae5ff915015b19c76016ebadd828d8 not found: ID does not exist" Oct 30 15:39:25.912874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.912855 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:39:25.926936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.926911 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:39:25.951042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951019 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:39:25.951328 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951309 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="init-config-reloader" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951328 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="init-config-reloader" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951342 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="config-reloader" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951348 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="config-reloader" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951365 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951370 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951381 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-web" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951385 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-web" Oct 30 15:39:25.951394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951395 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="prom-label-proxy" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951400 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="prom-label-proxy" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951408 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-metric" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951412 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-metric" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951421 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="alertmanager" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951425 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="alertmanager" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951483 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="prom-label-proxy" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951491 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-metric" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951498 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="alertmanager" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951504 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="config-reloader" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951513 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy" Oct 30 15:39:25.951697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.951526 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" containerName="kube-rbac-proxy-web" Oct 30 15:39:25.987807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.987784 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:39:25.987941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:25.987928 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.006457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006437 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 15:39:26.006640 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006626 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 15:39:26.006717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006641 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 15:39:26.006717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006659 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 15:39:26.006717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006644 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 15:39:26.006717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006663 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 15:39:26.006717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006668 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 15:39:26.006717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.006694 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cbhg8\"" Oct 30 15:39:26.020417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.020397 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 15:39:26.020519 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.020504 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 15:39:26.079547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079523 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-web-config\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079553 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079576 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/719797fd-5be6-4d5a-9af8-f2c72448cc11-config-out\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079593 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/719797fd-5be6-4d5a-9af8-f2c72448cc11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079665 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/719797fd-5be6-4d5a-9af8-f2c72448cc11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079742 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079784 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szctr\" (UniqueName: \"kubernetes.io/projected/719797fd-5be6-4d5a-9af8-f2c72448cc11-kube-api-access-szctr\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079808 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079885 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-config-volume\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.079959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079928 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.080039 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079963 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.080039 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.079978 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/719797fd-5be6-4d5a-9af8-f2c72448cc11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.080039 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.080001 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/719797fd-5be6-4d5a-9af8-f2c72448cc11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180443 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180472 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/719797fd-5be6-4d5a-9af8-f2c72448cc11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180491 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/719797fd-5be6-4d5a-9af8-f2c72448cc11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180511 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-web-config\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180528 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180547 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/719797fd-5be6-4d5a-9af8-f2c72448cc11-config-out\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180568 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/719797fd-5be6-4d5a-9af8-f2c72448cc11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180697 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/719797fd-5be6-4d5a-9af8-f2c72448cc11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180741 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180783 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szctr\" (UniqueName: \"kubernetes.io/projected/719797fd-5be6-4d5a-9af8-f2c72448cc11-kube-api-access-szctr\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180810 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180892 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-config-volume\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.180945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.180920 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.181753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.181711 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/719797fd-5be6-4d5a-9af8-f2c72448cc11-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.182381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.182357 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/719797fd-5be6-4d5a-9af8-f2c72448cc11-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.182541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.182521 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/719797fd-5be6-4d5a-9af8-f2c72448cc11-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.184186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.184160 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.184298 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.184194 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-web-config\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.184370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.184295 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.184788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.184510 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.184788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.184607 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/719797fd-5be6-4d5a-9af8-f2c72448cc11-config-out\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.184921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.184792 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/719797fd-5be6-4d5a-9af8-f2c72448cc11-tls-assets\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.184986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.184964 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.185084 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.185066 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-config-volume\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.185646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.185631 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/719797fd-5be6-4d5a-9af8-f2c72448cc11-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.206109 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.206091 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szctr\" (UniqueName: \"kubernetes.io/projected/719797fd-5be6-4d5a-9af8-f2c72448cc11-kube-api-access-szctr\") pod \"alertmanager-main-0\" (UID: \"719797fd-5be6-4d5a-9af8-f2c72448cc11\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.401311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.401254 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 15:39:26.735863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.735767 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 15:39:26.738646 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:39:26.738612 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719797fd_5be6_4d5a_9af8_f2c72448cc11.slice/crio-d23d30036039ac2e2e9c23564bf1c7ff389069e887219af63eaafe38e89c2bee WatchSource:0}: Error finding container d23d30036039ac2e2e9c23564bf1c7ff389069e887219af63eaafe38e89c2bee: Status 404 returned error can't find the container with id d23d30036039ac2e2e9c23564bf1c7ff389069e887219af63eaafe38e89c2bee Oct 30 15:39:26.840013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.839982 4401 generic.go:356] "Generic (PLEG): container finished" podID="719797fd-5be6-4d5a-9af8-f2c72448cc11" containerID="c5d0a2e5fe33d789939f804db112d3f5e1760525c848f475a2e655f7bd72b00f" exitCode=0 Oct 30 15:39:26.840435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.840019 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerDied","Data":"c5d0a2e5fe33d789939f804db112d3f5e1760525c848f475a2e655f7bd72b00f"} Oct 30 15:39:26.840435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:26.840067 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerStarted","Data":"d23d30036039ac2e2e9c23564bf1c7ff389069e887219af63eaafe38e89c2bee"} Oct 30 15:39:27.426232 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.426202 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2180244-21b7-47e8-a03d-a03320295f6a" path="/var/lib/kubelet/pods/a2180244-21b7-47e8-a03d-a03320295f6a/volumes" Oct 30 15:39:27.847493 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.847460 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerStarted","Data":"cc5935be318ecf3cbabb0fcd21f8a6038f21181031cf7874291c9523d5919a3f"} Oct 30 15:39:27.847914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.847528 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerStarted","Data":"e432f953f4efbea37a1dfcbbec6fe2c80ff7287daf70620c1d248b5d0d873a32"} Oct 30 15:39:27.847914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.847553 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerStarted","Data":"eff1434c391a36f4f8e3164d09d20c0543a59c2b901647fd3d33cf0b72c8c459"} Oct 30 15:39:27.847914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.847579 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerStarted","Data":"7897fb532d59af40548e542282466f7730769730decb55b32dc4a70261bf7359"} Oct 30 15:39:27.847914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.847601 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerStarted","Data":"2ed2fc64cb8918c5703ecff61c1ced7f9f5cac616b261c8fd041035a9ac7da17"} Oct 30 15:39:27.847914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.847623 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"719797fd-5be6-4d5a-9af8-f2c72448cc11","Type":"ContainerStarted","Data":"be7124c98fc6d7db5fb04fbb6722dbe745d2250626e7e496800e89e88a6c27fd"} Oct 30 15:39:27.918268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:27.918217 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.904699207 podStartE2EDuration="2.918204951s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.505744ms" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:27.917864514 +0000 UTC m=+76.617137670" watchObservedRunningTime="2026-10-30 15:39:27.918204951 +0000 UTC m=+76.617478142" Oct 30 15:39:33.644500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:33.644470 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:33.645060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:33.644521 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:33.649123 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:33.649105 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:33.874843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:33.874809 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b44548658-8dpd8" Oct 30 15:39:33.953234 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:33.953159 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5975b97954-2hglk"] Oct 30 15:39:51.964854 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:51.964803 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:39:58.976216 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:58.976175 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5975b97954-2hglk" podUID="faf41440-3225-4b97-ba2d-862c7cd5628b" containerName="console" containerID="cri-o://ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7" gracePeriod=15 Oct 30 15:39:59.209180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.209153 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:39:59.344936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.344849 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-console-config\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-console-config\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.344936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.344919 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faf41440-3225-4b97-ba2d-862c7cd5628b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/faf41440-3225-4b97-ba2d-862c7cd5628b-tmp\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.345163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.344951 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-service-ca\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-service-ca\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.345163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.344982 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-oauth-serving-cert\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.345163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345011 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-trusted-ca-bundle\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.345163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345059 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-serving-cert\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.345163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345102 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf41440-3225-4b97-ba2d-862c7cd5628b-kube-api-access-z5wfb\" (UniqueName: \"kubernetes.io/projected/faf41440-3225-4b97-ba2d-862c7cd5628b-kube-api-access-z5wfb\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.345387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345168 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-oauth-config\") pod \"faf41440-3225-4b97-ba2d-862c7cd5628b\" (UID: \"faf41440-3225-4b97-ba2d-862c7cd5628b\") " Oct 30 15:39:59.345387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345244 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faf41440-3225-4b97-ba2d-862c7cd5628b-tmp" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:59.345387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345285 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-console-config" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:59.345503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345435 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-oauth-serving-cert" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:59.345544 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345513 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-service-ca" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:59.345544 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345517 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-trusted-ca-bundle" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:59.345654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345567 4401 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-console-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.345654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345588 4401 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/faf41440-3225-4b97-ba2d-862c7cd5628b-tmp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.345654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.345600 4401 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-oauth-serving-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.347265 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.347236 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-serving-cert" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:59.347431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.347270 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-oauth-config" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:59.347431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.347287 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf41440-3225-4b97-ba2d-862c7cd5628b-kube-api-access-z5wfb" pod "faf41440-3225-4b97-ba2d-862c7cd5628b" (UID: "faf41440-3225-4b97-ba2d-862c7cd5628b"). InnerVolumeSpecName "kube-api-access-z5wfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:59.446241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.446212 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wfb\" (UniqueName: \"kubernetes.io/projected/faf41440-3225-4b97-ba2d-862c7cd5628b-kube-api-access-z5wfb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.446241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.446235 4401 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-oauth-config\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.446241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.446245 4401 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.446439 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.446254 4401 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/faf41440-3225-4b97-ba2d-862c7cd5628b-trusted-ca-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.446439 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.446264 4401 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/faf41440-3225-4b97-ba2d-862c7cd5628b-console-serving-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:39:59.965427 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.965392 4401 generic.go:356] "Generic (PLEG): container finished" podID="faf41440-3225-4b97-ba2d-862c7cd5628b" containerID="ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7" exitCode=0 Oct 30 15:39:59.965590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.965436 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5975b97954-2hglk" Oct 30 15:39:59.965590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.965479 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5975b97954-2hglk" event={"ID":"faf41440-3225-4b97-ba2d-862c7cd5628b","Type":"ContainerDied","Data":"ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7"} Oct 30 15:39:59.965590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.965529 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5975b97954-2hglk" event={"ID":"faf41440-3225-4b97-ba2d-862c7cd5628b","Type":"ContainerDied","Data":"1f8d3847ee24959ddc4787fb0e902093f6c70560c7d87ca2680d463d96ea91fc"} Oct 30 15:39:59.965590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.965550 4401 scope.go:122] "RemoveContainer" containerID="ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7" Oct 30 15:39:59.973366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.973341 4401 scope.go:122] "RemoveContainer" containerID="ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7" Oct 30 15:39:59.973644 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:39:59.973626 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7\": container with ID starting with ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7 not found: ID does not exist" containerID="ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7" Oct 30 15:39:59.973686 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:39:59.973652 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7"} err="failed to get container status \"ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7\": rpc error: code = NotFound desc = could not find container \"ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7\": container with ID starting with ab56eb49d4499006fdccb10452a011175f1aa8eb8abfeeb07bb509673d4fc1e7 not found: ID does not exist" Oct 30 15:40:00.022190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:40:00.022166 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5975b97954-2hglk"] Oct 30 15:40:00.036107 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:40:00.036087 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5975b97954-2hglk"] Oct 30 15:40:01.424653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:40:01.424623 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf41440-3225-4b97-ba2d-862c7cd5628b" path="/var/lib/kubelet/pods/faf41440-3225-4b97-ba2d-862c7cd5628b/volumes" Oct 30 15:40:21.967525 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:40:21.967486 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:40:51.964758 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:40:51.964682 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:41:21.964430 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:41:21.964394 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:41:51.965388 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:41:51.965349 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:41:57.751081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.750993 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9"] Oct 30 15:41:57.751657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.751304 4401 cpu_manager.go:414] "containerMap: removing container" podUID="faf41440-3225-4b97-ba2d-862c7cd5628b" containerName="console" Oct 30 15:41:57.751657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.751315 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf41440-3225-4b97-ba2d-862c7cd5628b" containerName="console" Oct 30 15:41:57.751657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.751396 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf41440-3225-4b97-ba2d-862c7cd5628b" containerName="console" Oct 30 15:41:57.753313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.753297 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.783885 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.783859 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 15:41:57.783885 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.783861 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 15:41:57.784098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.783912 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rq782\"" Oct 30 15:41:57.785031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.785007 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9"] Oct 30 15:41:57.819316 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.819289 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgzzf\" (UniqueName: \"kubernetes.io/projected/9f5deb57-53c4-4a0a-8327-8de790848998-kube-api-access-xgzzf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.819426 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.819327 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.819426 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.819351 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.919776 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.919749 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.919902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.919810 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzzf\" (UniqueName: \"kubernetes.io/projected/9f5deb57-53c4-4a0a-8327-8de790848998-kube-api-access-xgzzf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.919902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.919864 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.920134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.920114 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.920216 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.920140 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:57.946941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:57.946918 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzzf\" (UniqueName: \"kubernetes.io/projected/9f5deb57-53c4-4a0a-8327-8de790848998-kube-api-access-xgzzf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:58.064952 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:58.064896 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:41:58.362988 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:58.362965 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9"] Oct 30 15:41:58.365838 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:41:58.365797 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5deb57_53c4_4a0a_8327_8de790848998.slice/crio-a40b3caf6ad77bae1bb64b123eec32b35c79c679a2da707f58550e72199fbd54 WatchSource:0}: Error finding container a40b3caf6ad77bae1bb64b123eec32b35c79c679a2da707f58550e72199fbd54: Status 404 returned error can't find the container with id a40b3caf6ad77bae1bb64b123eec32b35c79c679a2da707f58550e72199fbd54 Oct 30 15:41:59.371865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:41:59.371815 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" event={"ID":"9f5deb57-53c4-4a0a-8327-8de790848998","Type":"ContainerStarted","Data":"a40b3caf6ad77bae1bb64b123eec32b35c79c679a2da707f58550e72199fbd54"} Oct 30 15:42:04.392293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:04.392257 4401 generic.go:356] "Generic (PLEG): container finished" podID="9f5deb57-53c4-4a0a-8327-8de790848998" containerID="3c6d2f77d99e2609ce7b66e8a33f15fbffb85f75c9accf548eedcabac8613f45" exitCode=0 Oct 30 15:42:04.392789 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:04.392324 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" event={"ID":"9f5deb57-53c4-4a0a-8327-8de790848998","Type":"ContainerDied","Data":"3c6d2f77d99e2609ce7b66e8a33f15fbffb85f75c9accf548eedcabac8613f45"} Oct 30 15:42:07.405641 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:07.405610 4401 generic.go:356] "Generic (PLEG): container finished" podID="9f5deb57-53c4-4a0a-8327-8de790848998" containerID="079545f67219566d2c6e4131d421075d5fef04de4d214b74f4508a04632f1349" exitCode=0 Oct 30 15:42:07.406070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:07.405700 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" event={"ID":"9f5deb57-53c4-4a0a-8327-8de790848998","Type":"ContainerDied","Data":"079545f67219566d2c6e4131d421075d5fef04de4d214b74f4508a04632f1349"} Oct 30 15:42:15.438811 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:15.438772 4401 generic.go:356] "Generic (PLEG): container finished" podID="9f5deb57-53c4-4a0a-8327-8de790848998" containerID="b62ad30ac93b6a323bf2558a9572016cd1f91af11e3c5288cc46e06551dc4db8" exitCode=0 Oct 30 15:42:15.439221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:15.438857 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" event={"ID":"9f5deb57-53c4-4a0a-8327-8de790848998","Type":"ContainerDied","Data":"b62ad30ac93b6a323bf2558a9572016cd1f91af11e3c5288cc46e06551dc4db8"} Oct 30 15:42:16.565633 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.565611 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:42:16.693367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.693305 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-bundle\") pod \"9f5deb57-53c4-4a0a-8327-8de790848998\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " Oct 30 15:42:16.693487 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.693370 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-util\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-util\") pod \"9f5deb57-53c4-4a0a-8327-8de790848998\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " Oct 30 15:42:16.693487 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.693399 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5deb57-53c4-4a0a-8327-8de790848998-kube-api-access-xgzzf\" (UniqueName: \"kubernetes.io/projected/9f5deb57-53c4-4a0a-8327-8de790848998-kube-api-access-xgzzf\") pod \"9f5deb57-53c4-4a0a-8327-8de790848998\" (UID: \"9f5deb57-53c4-4a0a-8327-8de790848998\") " Oct 30 15:42:16.693937 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.693902 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-bundle" pod "9f5deb57-53c4-4a0a-8327-8de790848998" (UID: "9f5deb57-53c4-4a0a-8327-8de790848998"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:16.695586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.695554 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5deb57-53c4-4a0a-8327-8de790848998-kube-api-access-xgzzf" pod "9f5deb57-53c4-4a0a-8327-8de790848998" (UID: "9f5deb57-53c4-4a0a-8327-8de790848998"). InnerVolumeSpecName "kube-api-access-xgzzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:16.698115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.698095 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-util" pod "9f5deb57-53c4-4a0a-8327-8de790848998" (UID: "9f5deb57-53c4-4a0a-8327-8de790848998"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:16.794688 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.794656 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgzzf\" (UniqueName: \"kubernetes.io/projected/9f5deb57-53c4-4a0a-8327-8de790848998-kube-api-access-xgzzf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:16.794688 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.794687 4401 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:16.794688 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:16.794696 4401 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f5deb57-53c4-4a0a-8327-8de790848998-util\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:17.447372 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:17.447341 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" event={"ID":"9f5deb57-53c4-4a0a-8327-8de790848998","Type":"ContainerDied","Data":"a40b3caf6ad77bae1bb64b123eec32b35c79c679a2da707f58550e72199fbd54"} Oct 30 15:42:17.447503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:17.447381 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cdxsv9" Oct 30 15:42:17.447503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:17.447390 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a40b3caf6ad77bae1bb64b123eec32b35c79c679a2da707f58550e72199fbd54" Oct 30 15:42:21.011169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011136 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz"] Oct 30 15:42:21.011584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011475 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9f5deb57-53c4-4a0a-8327-8de790848998" containerName="pull" Oct 30 15:42:21.011584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011492 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5deb57-53c4-4a0a-8327-8de790848998" containerName="pull" Oct 30 15:42:21.011584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011509 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9f5deb57-53c4-4a0a-8327-8de790848998" containerName="util" Oct 30 15:42:21.011584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011514 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5deb57-53c4-4a0a-8327-8de790848998" containerName="util" Oct 30 15:42:21.011584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011524 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9f5deb57-53c4-4a0a-8327-8de790848998" containerName="extract" Oct 30 15:42:21.011584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011529 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5deb57-53c4-4a0a-8327-8de790848998" containerName="extract" Oct 30 15:42:21.011789 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.011597 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5deb57-53c4-4a0a-8327-8de790848998" containerName="extract" Oct 30 15:42:21.054563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.054533 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz"] Oct 30 15:42:21.054725 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.054629 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.072289 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.072268 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 15:42:21.072481 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.072465 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-qhzkj\"" Oct 30 15:42:21.072527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.072470 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 15:42:21.233476 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.233447 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5248d00d-423a-4acf-9de7-a1aca736742b-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c7twz\" (UID: \"5248d00d-423a-4acf-9de7-a1aca736742b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.233639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.233489 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k752j\" (UniqueName: \"kubernetes.io/projected/5248d00d-423a-4acf-9de7-a1aca736742b-kube-api-access-k752j\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c7twz\" (UID: \"5248d00d-423a-4acf-9de7-a1aca736742b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.334934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.334861 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k752j\" (UniqueName: \"kubernetes.io/projected/5248d00d-423a-4acf-9de7-a1aca736742b-kube-api-access-k752j\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c7twz\" (UID: \"5248d00d-423a-4acf-9de7-a1aca736742b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.334934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.334931 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5248d00d-423a-4acf-9de7-a1aca736742b-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c7twz\" (UID: \"5248d00d-423a-4acf-9de7-a1aca736742b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.335271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.335256 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5248d00d-423a-4acf-9de7-a1aca736742b-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c7twz\" (UID: \"5248d00d-423a-4acf-9de7-a1aca736742b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.360440 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.360416 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k752j\" (UniqueName: \"kubernetes.io/projected/5248d00d-423a-4acf-9de7-a1aca736742b-kube-api-access-k752j\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-c7twz\" (UID: \"5248d00d-423a-4acf-9de7-a1aca736742b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.377384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.377362 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" Oct 30 15:42:21.726389 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:21.726334 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz"] Oct 30 15:42:21.728931 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:42:21.728904 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5248d00d_423a_4acf_9de7_a1aca736742b.slice/crio-2fcbb1bfa591bb186a402386ae421f4b3ec45a2efbb96db50935a6387174d7c2 WatchSource:0}: Error finding container 2fcbb1bfa591bb186a402386ae421f4b3ec45a2efbb96db50935a6387174d7c2: Status 404 returned error can't find the container with id 2fcbb1bfa591bb186a402386ae421f4b3ec45a2efbb96db50935a6387174d7c2 Oct 30 15:42:21.964874 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:42:21.964818 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:42:22.469612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:22.469576 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" event={"ID":"5248d00d-423a-4acf-9de7-a1aca736742b","Type":"ContainerStarted","Data":"2fcbb1bfa591bb186a402386ae421f4b3ec45a2efbb96db50935a6387174d7c2"} Oct 30 15:42:24.479913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:24.479878 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" event={"ID":"5248d00d-423a-4acf-9de7-a1aca736742b","Type":"ContainerStarted","Data":"cb4b2b52d5e897f7f70ba316de8293cc1977d6841ef6d7c0109b5be018fc65d6"} Oct 30 15:42:24.560712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:24.560656 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-c7twz" podStartSLOduration=2.655096089 podStartE2EDuration="4.560640691s" totalImagesPullingTime="1.905544602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:24.545386879 +0000 UTC m=+253.244660035" watchObservedRunningTime="2026-10-30 15:42:24.560640691 +0000 UTC m=+253.259913847" Oct 30 15:42:25.835894 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.835857 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk"] Oct 30 15:42:25.838348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.838327 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:25.855942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.855920 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 15:42:25.870022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.870001 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 15:42:25.870113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.870051 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-xfc5n\"" Oct 30 15:42:25.871302 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.871285 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk"] Oct 30 15:42:25.972396 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.972360 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmr6r\" (UniqueName: \"kubernetes.io/projected/a1a1bf1e-b578-4425-8be8-3157557433f4-kube-api-access-pmr6r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:25.972553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.972408 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:25.972553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:25.972546 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.073692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.073658 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.073903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.073699 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmr6r\" (UniqueName: \"kubernetes.io/projected/a1a1bf1e-b578-4425-8be8-3157557433f4-kube-api-access-pmr6r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.073903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.073726 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.074080 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.074062 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.074176 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.074155 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.099260 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.099227 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmr6r\" (UniqueName: \"kubernetes.io/projected/a1a1bf1e-b578-4425-8be8-3157557433f4-kube-api-access-pmr6r\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.150929 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.150907 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:26.447584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.447546 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk"] Oct 30 15:42:26.449690 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:42:26.449663 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a1bf1e_b578_4425_8be8_3157557433f4.slice/crio-a4a7485ccd06302fd3ca4f55e311bd6fc2c80fc5fa372a7480e8c07a537d8fed WatchSource:0}: Error finding container a4a7485ccd06302fd3ca4f55e311bd6fc2c80fc5fa372a7480e8c07a537d8fed: Status 404 returned error can't find the container with id a4a7485ccd06302fd3ca4f55e311bd6fc2c80fc5fa372a7480e8c07a537d8fed Oct 30 15:42:26.488373 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.488349 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" event={"ID":"a1a1bf1e-b578-4425-8be8-3157557433f4","Type":"ContainerStarted","Data":"a4a7485ccd06302fd3ca4f55e311bd6fc2c80fc5fa372a7480e8c07a537d8fed"} Oct 30 15:42:26.936874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.936819 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd"] Oct 30 15:42:26.939730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.939712 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:26.964664 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.964640 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd"] Oct 30 15:42:26.976871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.976844 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-9bmlq\"" Oct 30 15:42:26.976973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.976876 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 15:42:26.976973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.976916 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 15:42:26.979304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.979278 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzz4j\" (UniqueName: \"kubernetes.io/projected/88aa1a17-b1b7-4758-adef-7cf109e4f282-kube-api-access-dzz4j\") pod \"cert-manager-cainjector-5c57bbd4fc-w2ssd\" (UID: \"88aa1a17-b1b7-4758-adef-7cf109e4f282\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:26.979406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:26.979326 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88aa1a17-b1b7-4758-adef-7cf109e4f282-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w2ssd\" (UID: \"88aa1a17-b1b7-4758-adef-7cf109e4f282\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:27.080277 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.080244 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzz4j\" (UniqueName: \"kubernetes.io/projected/88aa1a17-b1b7-4758-adef-7cf109e4f282-kube-api-access-dzz4j\") pod \"cert-manager-cainjector-5c57bbd4fc-w2ssd\" (UID: \"88aa1a17-b1b7-4758-adef-7cf109e4f282\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:27.080407 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.080286 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88aa1a17-b1b7-4758-adef-7cf109e4f282-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w2ssd\" (UID: \"88aa1a17-b1b7-4758-adef-7cf109e4f282\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:27.105482 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.105456 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88aa1a17-b1b7-4758-adef-7cf109e4f282-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-w2ssd\" (UID: \"88aa1a17-b1b7-4758-adef-7cf109e4f282\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:27.105745 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.105724 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzz4j\" (UniqueName: \"kubernetes.io/projected/88aa1a17-b1b7-4758-adef-7cf109e4f282-kube-api-access-dzz4j\") pod \"cert-manager-cainjector-5c57bbd4fc-w2ssd\" (UID: \"88aa1a17-b1b7-4758-adef-7cf109e4f282\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:27.260012 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.259948 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" Oct 30 15:42:27.493763 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.493727 4401 generic.go:356] "Generic (PLEG): container finished" podID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerID="e993d39c4fcbce44596526b8c8eb68be434f28a6122d048ae5b96982d1f7672c" exitCode=0 Oct 30 15:42:27.493973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.493768 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" event={"ID":"a1a1bf1e-b578-4425-8be8-3157557433f4","Type":"ContainerDied","Data":"e993d39c4fcbce44596526b8c8eb68be434f28a6122d048ae5b96982d1f7672c"} Oct 30 15:42:27.617178 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:27.617154 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd"] Oct 30 15:42:27.619267 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:42:27.619239 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88aa1a17_b1b7_4758_adef_7cf109e4f282.slice/crio-3a9fcd8bce03699da7f1a8f682f5cbcd5c3e034a735a3e586e41f061f9999d37 WatchSource:0}: Error finding container 3a9fcd8bce03699da7f1a8f682f5cbcd5c3e034a735a3e586e41f061f9999d37: Status 404 returned error can't find the container with id 3a9fcd8bce03699da7f1a8f682f5cbcd5c3e034a735a3e586e41f061f9999d37 Oct 30 15:42:28.500149 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:28.500079 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" event={"ID":"88aa1a17-b1b7-4758-adef-7cf109e4f282","Type":"ContainerStarted","Data":"3a9fcd8bce03699da7f1a8f682f5cbcd5c3e034a735a3e586e41f061f9999d37"} Oct 30 15:42:29.506494 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:29.506462 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" event={"ID":"a1a1bf1e-b578-4425-8be8-3157557433f4","Type":"ContainerStarted","Data":"16690201da26c8f63e08f093385574ec1f89d29fb4d8c503e6bdd717aa085a16"} Oct 30 15:42:30.512159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:30.512125 4401 generic.go:356] "Generic (PLEG): container finished" podID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerID="16690201da26c8f63e08f093385574ec1f89d29fb4d8c503e6bdd717aa085a16" exitCode=0 Oct 30 15:42:30.512659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:30.512211 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" event={"ID":"a1a1bf1e-b578-4425-8be8-3157557433f4","Type":"ContainerDied","Data":"16690201da26c8f63e08f093385574ec1f89d29fb4d8c503e6bdd717aa085a16"} Oct 30 15:42:31.517604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:31.517569 4401 generic.go:356] "Generic (PLEG): container finished" podID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerID="c51d546b484c30b60018ccf9f83df8b1de69a9c798dc31e878295d079d2f4544" exitCode=0 Oct 30 15:42:31.518069 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:31.517618 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" event={"ID":"a1a1bf1e-b578-4425-8be8-3157557433f4","Type":"ContainerDied","Data":"c51d546b484c30b60018ccf9f83df8b1de69a9c798dc31e878295d079d2f4544"} Oct 30 15:42:32.641900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.641875 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:32.642724 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.642702 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29"] Oct 30 15:42:32.643064 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.643049 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerName="extract" Oct 30 15:42:32.643115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.643065 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerName="extract" Oct 30 15:42:32.643115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.643078 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerName="util" Oct 30 15:42:32.643115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.643083 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerName="util" Oct 30 15:42:32.643115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.643098 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerName="pull" Oct 30 15:42:32.643115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.643102 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerName="pull" Oct 30 15:42:32.643262 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.643162 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1a1bf1e-b578-4425-8be8-3157557433f4" containerName="extract" Oct 30 15:42:32.644947 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.644933 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:32.674645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.674619 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zfpf5\"" Oct 30 15:42:32.675809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.675791 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29"] Oct 30 15:42:32.716668 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.716643 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1a1bf1e-b578-4425-8be8-3157557433f4-kube-api-access-pmr6r\" (UniqueName: \"kubernetes.io/projected/a1a1bf1e-b578-4425-8be8-3157557433f4-kube-api-access-pmr6r\") pod \"a1a1bf1e-b578-4425-8be8-3157557433f4\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " Oct 30 15:42:32.716806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.716712 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-bundle\") pod \"a1a1bf1e-b578-4425-8be8-3157557433f4\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " Oct 30 15:42:32.716806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.716740 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-util\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-util\") pod \"a1a1bf1e-b578-4425-8be8-3157557433f4\" (UID: \"a1a1bf1e-b578-4425-8be8-3157557433f4\") " Oct 30 15:42:32.716954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.716933 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd5a7553-8377-498b-ab94-0fb53d0c83fb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-b7d29\" (UID: \"dd5a7553-8377-498b-ab94-0fb53d0c83fb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:32.717013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.716974 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rv4\" (UniqueName: \"kubernetes.io/projected/dd5a7553-8377-498b-ab94-0fb53d0c83fb-kube-api-access-r4rv4\") pod \"cert-manager-webhook-5dbbbf75cc-b7d29\" (UID: \"dd5a7553-8377-498b-ab94-0fb53d0c83fb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:32.717165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.717136 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-bundle" pod "a1a1bf1e-b578-4425-8be8-3157557433f4" (UID: "a1a1bf1e-b578-4425-8be8-3157557433f4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:32.718804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.718770 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1a1bf1e-b578-4425-8be8-3157557433f4-kube-api-access-pmr6r" pod "a1a1bf1e-b578-4425-8be8-3157557433f4" (UID: "a1a1bf1e-b578-4425-8be8-3157557433f4"). InnerVolumeSpecName "kube-api-access-pmr6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:32.721369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.721351 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-util" pod "a1a1bf1e-b578-4425-8be8-3157557433f4" (UID: "a1a1bf1e-b578-4425-8be8-3157557433f4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:32.818368 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.818303 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd5a7553-8377-498b-ab94-0fb53d0c83fb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-b7d29\" (UID: \"dd5a7553-8377-498b-ab94-0fb53d0c83fb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:32.818368 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.818339 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rv4\" (UniqueName: \"kubernetes.io/projected/dd5a7553-8377-498b-ab94-0fb53d0c83fb-kube-api-access-r4rv4\") pod \"cert-manager-webhook-5dbbbf75cc-b7d29\" (UID: \"dd5a7553-8377-498b-ab94-0fb53d0c83fb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:32.818550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.818442 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmr6r\" (UniqueName: \"kubernetes.io/projected/a1a1bf1e-b578-4425-8be8-3157557433f4-kube-api-access-pmr6r\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:32.818550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.818458 4401 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:32.818550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.818472 4401 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1a1bf1e-b578-4425-8be8-3157557433f4-util\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:32.844914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.844885 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dd5a7553-8377-498b-ab94-0fb53d0c83fb-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-b7d29\" (UID: \"dd5a7553-8377-498b-ab94-0fb53d0c83fb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:32.845151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.845129 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rv4\" (UniqueName: \"kubernetes.io/projected/dd5a7553-8377-498b-ab94-0fb53d0c83fb-kube-api-access-r4rv4\") pod \"cert-manager-webhook-5dbbbf75cc-b7d29\" (UID: \"dd5a7553-8377-498b-ab94-0fb53d0c83fb\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:32.955405 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:32.955369 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:33.257814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:33.257788 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29"] Oct 30 15:42:33.260246 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:42:33.260221 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd5a7553_8377_498b_ab94_0fb53d0c83fb.slice/crio-8c3cd5ccd57cd5bba52174bf44f8771c02fad9527b99af4169b68a55713ad4c2 WatchSource:0}: Error finding container 8c3cd5ccd57cd5bba52174bf44f8771c02fad9527b99af4169b68a55713ad4c2: Status 404 returned error can't find the container with id 8c3cd5ccd57cd5bba52174bf44f8771c02fad9527b99af4169b68a55713ad4c2 Oct 30 15:42:33.527754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:33.527675 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" Oct 30 15:42:33.527939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:33.527676 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85w8zqk" event={"ID":"a1a1bf1e-b578-4425-8be8-3157557433f4","Type":"ContainerDied","Data":"a4a7485ccd06302fd3ca4f55e311bd6fc2c80fc5fa372a7480e8c07a537d8fed"} Oct 30 15:42:33.527939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:33.527808 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4a7485ccd06302fd3ca4f55e311bd6fc2c80fc5fa372a7480e8c07a537d8fed" Oct 30 15:42:33.529059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:33.529025 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" event={"ID":"dd5a7553-8377-498b-ab94-0fb53d0c83fb","Type":"ContainerStarted","Data":"8c3cd5ccd57cd5bba52174bf44f8771c02fad9527b99af4169b68a55713ad4c2"} Oct 30 15:42:36.545113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:36.545075 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" event={"ID":"dd5a7553-8377-498b-ab94-0fb53d0c83fb","Type":"ContainerStarted","Data":"301884ccafdc09380f9f9b85b0a53750f5c81671483e0a813f0bd87df5490170"} Oct 30 15:42:36.545558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:36.545200 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:36.589844 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:36.589786 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" podStartSLOduration=1.9316210310000002 podStartE2EDuration="4.58977326s" totalImagesPullingTime="2.658152229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:36.588842769 +0000 UTC m=+265.288115922" watchObservedRunningTime="2026-10-30 15:42:36.58977326 +0000 UTC m=+265.289046416" Oct 30 15:42:37.550693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:37.550645 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" event={"ID":"88aa1a17-b1b7-4758-adef-7cf109e4f282","Type":"ContainerStarted","Data":"36f4dde4dc50a6d92dc99a0dcb6fd3a05d7d2590731d5a8ee250a5763ecf6e70"} Oct 30 15:42:37.599526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:37.596545 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-w2ssd" podStartSLOduration=2.590075793 podStartE2EDuration="11.59652728s" totalImagesPullingTime="9.006451487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:37.593731083 +0000 UTC m=+266.293004249" watchObservedRunningTime="2026-10-30 15:42:37.59652728 +0000 UTC m=+266.295800436" Oct 30 15:42:42.552694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:42.552665 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" Oct 30 15:42:45.305725 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.305688 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-vqtvh"] Oct 30 15:42:45.307938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.307922 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.326508 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.326477 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vqtvh"] Oct 30 15:42:45.339015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.338993 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tlct2\"" Oct 30 15:42:45.419007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.418979 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmxc7\" (UniqueName: \"kubernetes.io/projected/df664835-4209-45b4-baaa-cbe7eff27d1b-kube-api-access-qmxc7\") pod \"cert-manager-597b95cb84-vqtvh\" (UID: \"df664835-4209-45b4-baaa-cbe7eff27d1b\") " pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.419175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.419044 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df664835-4209-45b4-baaa-cbe7eff27d1b-bound-sa-token\") pod \"cert-manager-597b95cb84-vqtvh\" (UID: \"df664835-4209-45b4-baaa-cbe7eff27d1b\") " pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.519604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.519571 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df664835-4209-45b4-baaa-cbe7eff27d1b-bound-sa-token\") pod \"cert-manager-597b95cb84-vqtvh\" (UID: \"df664835-4209-45b4-baaa-cbe7eff27d1b\") " pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.519741 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.519644 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmxc7\" (UniqueName: \"kubernetes.io/projected/df664835-4209-45b4-baaa-cbe7eff27d1b-kube-api-access-qmxc7\") pod \"cert-manager-597b95cb84-vqtvh\" (UID: \"df664835-4209-45b4-baaa-cbe7eff27d1b\") " pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.547241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.547215 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df664835-4209-45b4-baaa-cbe7eff27d1b-bound-sa-token\") pod \"cert-manager-597b95cb84-vqtvh\" (UID: \"df664835-4209-45b4-baaa-cbe7eff27d1b\") " pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.547369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.547307 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmxc7\" (UniqueName: \"kubernetes.io/projected/df664835-4209-45b4-baaa-cbe7eff27d1b-kube-api-access-qmxc7\") pod \"cert-manager-597b95cb84-vqtvh\" (UID: \"df664835-4209-45b4-baaa-cbe7eff27d1b\") " pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.617887 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.617861 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vqtvh" Oct 30 15:42:45.916792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:45.916767 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vqtvh"] Oct 30 15:42:45.919393 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:42:45.919365 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf664835_4209_45b4_baaa_cbe7eff27d1b.slice/crio-966a2f8e68a62efd8b5307dd17fa47a5af27b05f36c9e9aaba64cb25ab033eb1 WatchSource:0}: Error finding container 966a2f8e68a62efd8b5307dd17fa47a5af27b05f36c9e9aaba64cb25ab033eb1: Status 404 returned error can't find the container with id 966a2f8e68a62efd8b5307dd17fa47a5af27b05f36c9e9aaba64cb25ab033eb1 Oct 30 15:42:46.587450 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:46.587414 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vqtvh" event={"ID":"df664835-4209-45b4-baaa-cbe7eff27d1b","Type":"ContainerStarted","Data":"0c257c44a8d3fc100d444a42ab0929c35a7d45c6ab533f9f96576d9c2112f12f"} Oct 30 15:42:46.587809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:46.587461 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vqtvh" event={"ID":"df664835-4209-45b4-baaa-cbe7eff27d1b","Type":"ContainerStarted","Data":"966a2f8e68a62efd8b5307dd17fa47a5af27b05f36c9e9aaba64cb25ab033eb1"} Oct 30 15:42:46.631100 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:46.631055 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-vqtvh" podStartSLOduration=1.631042434 podStartE2EDuration="1.631042434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:46.630366199 +0000 UTC m=+275.329639357" watchObservedRunningTime="2026-10-30 15:42:46.631042434 +0000 UTC m=+275.330315591" Oct 30 15:42:51.965572 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:42:51.965536 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:42:54.237469 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.237434 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg"] Oct 30 15:42:54.239833 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.239808 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.271818 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.271790 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-t672x\"" Oct 30 15:42:54.272013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.271985 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 15:42:54.272072 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.272037 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 15:42:54.273238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.273217 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg"] Oct 30 15:42:54.298662 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.298629 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-747vl\" (UniqueName: \"kubernetes.io/projected/79b637a3-82d7-4f96-a8ed-f7fd894ac322-kube-api-access-747vl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.298799 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.298730 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.298877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.298861 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.400281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.400239 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747vl\" (UniqueName: \"kubernetes.io/projected/79b637a3-82d7-4f96-a8ed-f7fd894ac322-kube-api-access-747vl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.400490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.400288 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.400490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.400347 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.400749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.400728 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.400809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.400766 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.425949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.425915 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-747vl\" (UniqueName: \"kubernetes.io/projected/79b637a3-82d7-4f96-a8ed-f7fd894ac322-kube-api-access-747vl\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.552922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.552842 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:54.855489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:54.855441 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg"] Oct 30 15:42:54.857557 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:42:54.857522 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b637a3_82d7_4f96_a8ed_f7fd894ac322.slice/crio-876e2ff675cda027fd6914f6151f4fddafe2991852190f63e205538d389bc534 WatchSource:0}: Error finding container 876e2ff675cda027fd6914f6151f4fddafe2991852190f63e205538d389bc534: Status 404 returned error can't find the container with id 876e2ff675cda027fd6914f6151f4fddafe2991852190f63e205538d389bc534 Oct 30 15:42:55.624902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:55.624871 4401 generic.go:356] "Generic (PLEG): container finished" podID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerID="d2201f856e30706625c76ded57de380e96cd00c32cb8e0b3b850dc7f638ff8f2" exitCode=0 Oct 30 15:42:55.625249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:55.624950 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" event={"ID":"79b637a3-82d7-4f96-a8ed-f7fd894ac322","Type":"ContainerDied","Data":"d2201f856e30706625c76ded57de380e96cd00c32cb8e0b3b850dc7f638ff8f2"} Oct 30 15:42:55.625249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:55.625000 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" event={"ID":"79b637a3-82d7-4f96-a8ed-f7fd894ac322","Type":"ContainerStarted","Data":"876e2ff675cda027fd6914f6151f4fddafe2991852190f63e205538d389bc534"} Oct 30 15:42:56.631730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:56.631689 4401 generic.go:356] "Generic (PLEG): container finished" podID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerID="aa78f862a5801cd0cfa484bb2cd6e1f778f067d576a9c0aed31d1414efb3647b" exitCode=0 Oct 30 15:42:56.632126 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:56.631767 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" event={"ID":"79b637a3-82d7-4f96-a8ed-f7fd894ac322","Type":"ContainerDied","Data":"aa78f862a5801cd0cfa484bb2cd6e1f778f067d576a9c0aed31d1414efb3647b"} Oct 30 15:42:57.638571 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:57.638539 4401 generic.go:356] "Generic (PLEG): container finished" podID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerID="43f7cd521f4d11a0647b72c480d5aeac52ce96bae96f22704013daef5459d3ef" exitCode=0 Oct 30 15:42:57.638976 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:57.638621 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" event={"ID":"79b637a3-82d7-4f96-a8ed-f7fd894ac322","Type":"ContainerDied","Data":"43f7cd521f4d11a0647b72c480d5aeac52ce96bae96f22704013daef5459d3ef"} Oct 30 15:42:58.770527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.770502 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:58.837523 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.837486 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b637a3-82d7-4f96-a8ed-f7fd894ac322-kube-api-access-747vl\" (UniqueName: \"kubernetes.io/projected/79b637a3-82d7-4f96-a8ed-f7fd894ac322-kube-api-access-747vl\") pod \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " Oct 30 15:42:58.837678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.837564 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-bundle\") pod \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " Oct 30 15:42:58.837678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.837586 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-util\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-util\") pod \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\" (UID: \"79b637a3-82d7-4f96-a8ed-f7fd894ac322\") " Oct 30 15:42:58.838212 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.838179 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-bundle" pod "79b637a3-82d7-4f96-a8ed-f7fd894ac322" (UID: "79b637a3-82d7-4f96-a8ed-f7fd894ac322"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:58.839602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.839572 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b637a3-82d7-4f96-a8ed-f7fd894ac322-kube-api-access-747vl" pod "79b637a3-82d7-4f96-a8ed-f7fd894ac322" (UID: "79b637a3-82d7-4f96-a8ed-f7fd894ac322"). InnerVolumeSpecName "kube-api-access-747vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:58.843227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.843203 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-util" pod "79b637a3-82d7-4f96-a8ed-f7fd894ac322" (UID: "79b637a3-82d7-4f96-a8ed-f7fd894ac322"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:58.938880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.938792 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-747vl\" (UniqueName: \"kubernetes.io/projected/79b637a3-82d7-4f96-a8ed-f7fd894ac322-kube-api-access-747vl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:58.938880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.938852 4401 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:58.938880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:58.938861 4401 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/79b637a3-82d7-4f96-a8ed-f7fd894ac322-util\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:42:59.648675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:59.648652 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" Oct 30 15:42:59.648865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:59.648684 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71rdckg" event={"ID":"79b637a3-82d7-4f96-a8ed-f7fd894ac322","Type":"ContainerDied","Data":"876e2ff675cda027fd6914f6151f4fddafe2991852190f63e205538d389bc534"} Oct 30 15:42:59.648865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:42:59.648732 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="876e2ff675cda027fd6914f6151f4fddafe2991852190f63e205538d389bc534" Oct 30 15:43:05.635671 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.635641 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7"] Oct 30 15:43:05.636125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.635991 4401 cpu_manager.go:414] "containerMap: removing container" podUID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerName="pull" Oct 30 15:43:05.636125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.636002 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerName="pull" Oct 30 15:43:05.636125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.636016 4401 cpu_manager.go:414] "containerMap: removing container" podUID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerName="extract" Oct 30 15:43:05.636125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.636021 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerName="extract" Oct 30 15:43:05.636125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.636034 4401 cpu_manager.go:414] "containerMap: removing container" podUID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerName="util" Oct 30 15:43:05.636125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.636039 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerName="util" Oct 30 15:43:05.636125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.636095 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b637a3-82d7-4f96-a8ed-f7fd894ac322" containerName="extract" Oct 30 15:43:05.640517 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.640498 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.678035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.678006 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7"] Oct 30 15:43:05.726069 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.726042 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 15:43:05.726326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.726310 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 15:43:05.726421 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.726403 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 15:43:05.726981 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.726960 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gqp9p\"" Oct 30 15:43:05.728509 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.728495 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 15:43:05.728567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.728495 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 15:43:05.800729 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.800703 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.800853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.800734 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce734ad7-04f7-4520-a0f7-3710d393736a-tmp\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.800853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.800775 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce734ad7-04f7-4520-a0f7-3710d393736a-manifests\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.800853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.800818 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.800985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.800943 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6qww\" (UniqueName: \"kubernetes.io/projected/ce734ad7-04f7-4520-a0f7-3710d393736a-kube-api-access-m6qww\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.801032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.800979 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.902227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.902139 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.902227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.902176 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce734ad7-04f7-4520-a0f7-3710d393736a-tmp\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.902227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.902214 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce734ad7-04f7-4520-a0f7-3710d393736a-manifests\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.902518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.902318 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.902518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.902400 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6qww\" (UniqueName: \"kubernetes.io/projected/ce734ad7-04f7-4520-a0f7-3710d393736a-kube-api-access-m6qww\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.902518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.902436 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.902816 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.902746 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce734ad7-04f7-4520-a0f7-3710d393736a-manifests\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.904580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.904556 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce734ad7-04f7-4520-a0f7-3710d393736a-tmp\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.904851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.904816 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.904966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.904935 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.905037 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.905009 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce734ad7-04f7-4520-a0f7-3710d393736a-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.942733 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.942708 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6qww\" (UniqueName: \"kubernetes.io/projected/ce734ad7-04f7-4520-a0f7-3710d393736a-kube-api-access-m6qww\") pod \"opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7\" (UID: \"ce734ad7-04f7-4520-a0f7-3710d393736a\") " pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:05.952382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:05.952363 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:06.341432 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:43:06.341403 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce734ad7_04f7_4520_a0f7_3710d393736a.slice/crio-8b3e67f51d2963813e5be4086cee5b9428e95c3739c22aa697c6b06fa9887268 WatchSource:0}: Error finding container 8b3e67f51d2963813e5be4086cee5b9428e95c3739c22aa697c6b06fa9887268: Status 404 returned error can't find the container with id 8b3e67f51d2963813e5be4086cee5b9428e95c3739c22aa697c6b06fa9887268 Oct 30 15:43:06.343592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:06.343571 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7"] Oct 30 15:43:06.677471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:06.677434 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" event={"ID":"ce734ad7-04f7-4520-a0f7-3710d393736a","Type":"ContainerStarted","Data":"8b3e67f51d2963813e5be4086cee5b9428e95c3739c22aa697c6b06fa9887268"} Oct 30 15:43:09.692998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:09.692957 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" event={"ID":"ce734ad7-04f7-4520-a0f7-3710d393736a","Type":"ContainerStarted","Data":"1faf283c64eb5f65923fc342ed062c69de78df4765d4463784dd85aa90ba8327"} Oct 30 15:43:10.698355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:10.698319 4401 generic.go:356] "Generic (PLEG): container finished" podID="ce734ad7-04f7-4520-a0f7-3710d393736a" containerID="1faf283c64eb5f65923fc342ed062c69de78df4765d4463784dd85aa90ba8327" exitCode=0 Oct 30 15:43:10.698809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:10.698408 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" event={"ID":"ce734ad7-04f7-4520-a0f7-3710d393736a","Type":"ContainerDied","Data":"1faf283c64eb5f65923fc342ed062c69de78df4765d4463784dd85aa90ba8327"} Oct 30 15:43:11.703765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:11.703729 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" event={"ID":"ce734ad7-04f7-4520-a0f7-3710d393736a","Type":"ContainerStarted","Data":"214e54143c7149c4ccdcf97cad1d942d9c29f8dd17718f8f28290ae641cd0539"} Oct 30 15:43:11.704163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:11.703867 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:11.829638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:11.829584 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" podStartSLOduration=3.165131651 podStartE2EDuration="6.829571085s" totalImagesPullingTime="3.165072374s" totalInitContainerRuntime="499.36706ms" isStatefulPod=false podCreationTimestamp="2026-10-30 15:43:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:43:11.814331628 +0000 UTC m=+300.513604785" watchObservedRunningTime="2026-10-30 15:43:11.829571085 +0000 UTC m=+300.528844242" Oct 30 15:43:21.965125 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:43:21.965077 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:43:22.709690 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:22.709664 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7" Oct 30 15:43:26.370969 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.370891 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx"] Oct 30 15:43:26.374141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.374124 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.394171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.394140 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 15:43:26.394349 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.394332 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cxkx5\"" Oct 30 15:43:26.408206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.408177 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 15:43:26.408318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.408235 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 15:43:26.408318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.408276 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 15:43:26.410390 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.410360 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx"] Oct 30 15:43:26.469990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.469963 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1b87c2-0b2c-437b-95db-76f18662d056-tls-certs\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.470134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.469995 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1b87c2-0b2c-437b-95db-76f18662d056-tmp\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.470134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.470039 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2tzv\" (UniqueName: \"kubernetes.io/projected/1d1b87c2-0b2c-437b-95db-76f18662d056-kube-api-access-j2tzv\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.571138 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.571106 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2tzv\" (UniqueName: \"kubernetes.io/projected/1d1b87c2-0b2c-437b-95db-76f18662d056-kube-api-access-j2tzv\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.571269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.571171 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1b87c2-0b2c-437b-95db-76f18662d056-tls-certs\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.571269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.571194 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1b87c2-0b2c-437b-95db-76f18662d056-tmp\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.573471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.573442 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1b87c2-0b2c-437b-95db-76f18662d056-tmp\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.573648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.573630 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1b87c2-0b2c-437b-95db-76f18662d056-tls-certs\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.600464 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.600441 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2tzv\" (UniqueName: \"kubernetes.io/projected/1d1b87c2-0b2c-437b-95db-76f18662d056-kube-api-access-j2tzv\") pod \"kube-auth-proxy-6f6967bb6d-fjrbx\" (UID: \"1d1b87c2-0b2c-437b-95db-76f18662d056\") " pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.686214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.686144 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" Oct 30 15:43:26.991910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.991883 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx"] Oct 30 15:43:26.993642 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:43:26.993621 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1b87c2_0b2c_437b_95db_76f18662d056.slice/crio-fd444170afce17f732d391441b61fc530ca8f4d320ca41634bbbdd4dc1dd2411 WatchSource:0}: Error finding container fd444170afce17f732d391441b61fc530ca8f4d320ca41634bbbdd4dc1dd2411: Status 404 returned error can't find the container with id fd444170afce17f732d391441b61fc530ca8f4d320ca41634bbbdd4dc1dd2411 Oct 30 15:43:26.995518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:26.995503 4401 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:43:27.776884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:27.776840 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" event={"ID":"1d1b87c2-0b2c-437b-95db-76f18662d056","Type":"ContainerStarted","Data":"fd444170afce17f732d391441b61fc530ca8f4d320ca41634bbbdd4dc1dd2411"} Oct 30 15:43:28.434654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.434619 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5"] Oct 30 15:43:28.443398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.443368 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.475300 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.475268 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 15:43:28.475518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.475493 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 15:43:28.475518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.475508 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bvrzq\"" Oct 30 15:43:28.475518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.475495 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 15:43:28.475727 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.475507 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 15:43:28.477675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.477654 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5"] Oct 30 15:43:28.484368 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.484343 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.484452 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.484411 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.484634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.484615 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d728b-ffb8-4845-94f2-00ca8968d82a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.484695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.484652 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.484758 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.484740 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhshv\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-kube-api-access-fhshv\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.484815 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.484776 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.485004 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.484981 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.585417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.585383 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.585613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.585431 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.585613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.585464 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.585613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.585502 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d728b-ffb8-4845-94f2-00ca8968d82a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.585613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.585530 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.585613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.585570 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhshv\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-kube-api-access-fhshv\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.585613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.585602 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.586534 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.586506 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.588273 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.588249 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.588430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.588405 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.588489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.588427 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d728b-ffb8-4845-94f2-00ca8968d82a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.588489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.588426 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.612173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.612097 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhshv\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-kube-api-access-fhshv\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.626069 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.626046 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mdlr5\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:28.759257 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:28.759175 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:29.092109 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:29.092083 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5"] Oct 30 15:43:29.093732 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:43:29.093697 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520d728b_ffb8_4845_94f2_00ca8968d82a.slice/crio-c71542ad0e5e1945e0fc9458ee7b7eba32981f9fe7b5ce63e207ad53f86603d4 WatchSource:0}: Error finding container c71542ad0e5e1945e0fc9458ee7b7eba32981f9fe7b5ce63e207ad53f86603d4: Status 404 returned error can't find the container with id c71542ad0e5e1945e0fc9458ee7b7eba32981f9fe7b5ce63e207ad53f86603d4 Oct 30 15:43:29.788248 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:29.788216 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" event={"ID":"1d1b87c2-0b2c-437b-95db-76f18662d056","Type":"ContainerStarted","Data":"7a96b1aac452ca92ee81918cf7a207994bea280d80d1e844ef32997c539aa7fb"} Oct 30 15:43:29.789635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:29.789610 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" event={"ID":"520d728b-ffb8-4845-94f2-00ca8968d82a","Type":"ContainerStarted","Data":"c71542ad0e5e1945e0fc9458ee7b7eba32981f9fe7b5ce63e207ad53f86603d4"} Oct 30 15:43:31.656621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:31.656590 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:43:31.656879 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:31.656654 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:43:31.800070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:31.800036 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" event={"ID":"520d728b-ffb8-4845-94f2-00ca8968d82a","Type":"ContainerStarted","Data":"dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53"} Oct 30 15:43:31.800254 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:31.800135 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:31.877632 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:31.877581 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f6967bb6d-fjrbx" podStartSLOduration=3.529469459 podStartE2EDuration="5.877567437s" totalImagesPullingTime="2.348097978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:43:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:43:29.837494154 +0000 UTC m=+318.536767312" watchObservedRunningTime="2026-10-30 15:43:31.877567437 +0000 UTC m=+320.576840594" Oct 30 15:43:32.805671 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:32.805642 4401 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-mdlr5 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:43:32.806065 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:32.805691 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" podUID="520d728b-ffb8-4845-94f2-00ca8968d82a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:43:33.475127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.475072 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" podStartSLOduration=2.915438575 podStartE2EDuration="5.475058407s" totalImagesPullingTime="2.559619832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:43:31.889978492 +0000 UTC m=+320.589251649" watchObservedRunningTime="2026-10-30 15:43:33.475058407 +0000 UTC m=+322.174331571" Oct 30 15:43:33.475676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.475660 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m"] Oct 30 15:43:33.479228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.479207 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.508558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.508531 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m"] Oct 30 15:43:33.527087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527055 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527215 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527138 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527215 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527188 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527322 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527216 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527322 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527316 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527349 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527375 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpdd\" (UniqueName: \"kubernetes.io/projected/75b74b6b-5966-4dbb-9e14-92c95de9d211-kube-api-access-klpdd\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527410 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b74b6b-5966-4dbb-9e14-92c95de9d211-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.527535 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.527438 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.545964 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.545934 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-p5npz\"" Oct 30 15:43:33.628791 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.628754 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b74b6b-5966-4dbb-9e14-92c95de9d211-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.628791 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.628793 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.628857 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.628882 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.628905 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.628942 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.628995 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.629027 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.629347 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpdd\" (UniqueName: \"kubernetes.io/projected/75b74b6b-5966-4dbb-9e14-92c95de9d211-kube-api-access-klpdd\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.629417 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.629359 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.629461 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629631 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.629515 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.629680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.629633 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b74b6b-5966-4dbb-9e14-92c95de9d211-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.631348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.631327 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.631580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.631562 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.655016 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.654988 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b74b6b-5966-4dbb-9e14-92c95de9d211-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.655145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.655127 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpdd\" (UniqueName: \"kubernetes.io/projected/75b74b6b-5966-4dbb-9e14-92c95de9d211-kube-api-access-klpdd\") pod \"data-science-gateway-data-science-gateway-class-5c79575b6f29t5m\" (UID: \"75b74b6b-5966-4dbb-9e14-92c95de9d211\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:33.792277 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:33.792201 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:34.114514 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:34.114483 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m"] Oct 30 15:43:34.115488 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:43:34.115455 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b74b6b_5966_4dbb_9e14_92c95de9d211.slice/crio-a814c6e3db5471124fd3463f49c627fb2ec7f249ee59fc7b0a9e3280dc19c7ce WatchSource:0}: Error finding container a814c6e3db5471124fd3463f49c627fb2ec7f249ee59fc7b0a9e3280dc19c7ce: Status 404 returned error can't find the container with id a814c6e3db5471124fd3463f49c627fb2ec7f249ee59fc7b0a9e3280dc19c7ce Oct 30 15:43:34.818069 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:34.818030 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" event={"ID":"75b74b6b-5966-4dbb-9e14-92c95de9d211","Type":"ContainerStarted","Data":"a814c6e3db5471124fd3463f49c627fb2ec7f249ee59fc7b0a9e3280dc19c7ce"} Oct 30 15:43:35.805391 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:35.805360 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:43:36.849154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:36.849120 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:43:36.849430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:36.849188 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:43:36.849430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:36.849220 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:43:37.839547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:37.839509 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" event={"ID":"75b74b6b-5966-4dbb-9e14-92c95de9d211","Type":"ContainerStarted","Data":"68037d8be8f956b0c22744303b5046e389f6238212003a84bb57bcfcab2f6e44"} Oct 30 15:43:37.902743 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:37.902674 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" podStartSLOduration=2.171920365 podStartE2EDuration="4.902654878s" totalImagesPullingTime="2.730734513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:43:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:43:37.901650766 +0000 UTC m=+326.600923936" watchObservedRunningTime="2026-10-30 15:43:37.902654878 +0000 UTC m=+326.601928036" Oct 30 15:43:38.792762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:38.792724 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:38.797449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:38.797424 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:38.843799 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:38.843769 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:38.844573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:43:38.844556 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5c79575b6f29t5m" Oct 30 15:43:51.964687 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:43:51.964650 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:44:21.964794 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:44:21.964755 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:44:31.640386 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.640354 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr"] Oct 30 15:44:31.643647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.643626 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.666632 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.666609 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr"] Oct 30 15:44:31.723567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.723526 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.723567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.723567 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4958a-0741-47b7-a199-9e51f9cf2e9a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.723748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.723645 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.723748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.723700 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.723748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.723721 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.723884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.723792 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.723884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.723877 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hs5j\" (UniqueName: \"kubernetes.io/projected/afd4958a-0741-47b7-a199-9e51f9cf2e9a-kube-api-access-8hs5j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825019 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.824986 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825019 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.825020 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4958a-0741-47b7-a199-9e51f9cf2e9a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825233 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.825164 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825291 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.825262 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.825303 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.825427 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825689 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.825666 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hs5j\" (UniqueName: \"kubernetes.io/projected/afd4958a-0741-47b7-a199-9e51f9cf2e9a-kube-api-access-8hs5j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.825817 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.825789 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.827339 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.827317 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/afd4958a-0741-47b7-a199-9e51f9cf2e9a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.827599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.827576 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.827865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.827846 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.827940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.827890 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/afd4958a-0741-47b7-a199-9e51f9cf2e9a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.858749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.858719 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/afd4958a-0741-47b7-a199-9e51f9cf2e9a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.860167 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.860146 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hs5j\" (UniqueName: \"kubernetes.io/projected/afd4958a-0741-47b7-a199-9e51f9cf2e9a-kube-api-access-8hs5j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-jndpr\" (UID: \"afd4958a-0741-47b7-a199-9e51f9cf2e9a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:31.955153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:31.955095 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:32.283337 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:32.283309 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr"] Oct 30 15:44:32.296961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:32.296912 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:44:32.297231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:32.297199 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:44:33.090920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:33.090881 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" event={"ID":"afd4958a-0741-47b7-a199-9e51f9cf2e9a","Type":"ContainerStarted","Data":"932f347415f3a04a4d257784eb9abcd6fcd655517f781b5181105efa9cdbefd8"} Oct 30 15:44:33.091355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:33.091059 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:33.091355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:33.091096 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" event={"ID":"afd4958a-0741-47b7-a199-9e51f9cf2e9a","Type":"ContainerStarted","Data":"90a8b948061499a5c7a45f39e4a057c5c7d3b5cfc47a66c9375c32acd068c137"} Oct 30 15:44:33.092361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:33.092336 4401 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-jndpr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:44:33.092474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:33.092376 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" podUID="afd4958a-0741-47b7-a199-9e51f9cf2e9a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:44:33.142540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:33.142497 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" podStartSLOduration=2.142484836 podStartE2EDuration="2.142484836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:33.140671335 +0000 UTC m=+381.839944491" watchObservedRunningTime="2026-10-30 15:44:33.142484836 +0000 UTC m=+381.841757993" Oct 30 15:44:34.095545 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.095515 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-jndpr" Oct 30 15:44:34.212563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.212536 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5"] Oct 30 15:44:34.212736 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.212706 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" podUID="520d728b-ffb8-4845-94f2-00ca8968d82a" containerName="discovery" containerID="cri-o://dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53" gracePeriod=30 Oct 30 15:44:34.455008 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.454982 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:44:34.548436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548405 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-dns-cert\") pod \"520d728b-ffb8-4845-94f2-00ca8968d82a\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " Oct 30 15:44:34.548635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548461 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-ca-configmap\") pod \"520d728b-ffb8-4845-94f2-00ca8968d82a\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " Oct 30 15:44:34.548635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548522 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-cacerts\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-cacerts\") pod \"520d728b-ffb8-4845-94f2-00ca8968d82a\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " Oct 30 15:44:34.548635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548552 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-kube-api-access-fhshv\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-kube-api-access-fhshv\") pod \"520d728b-ffb8-4845-94f2-00ca8968d82a\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " Oct 30 15:44:34.548635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548608 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-token\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-token\") pod \"520d728b-ffb8-4845-94f2-00ca8968d82a\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " Oct 30 15:44:34.548878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548636 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520d728b-ffb8-4845-94f2-00ca8968d82a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d728b-ffb8-4845-94f2-00ca8968d82a-local-certs\") pod \"520d728b-ffb8-4845-94f2-00ca8968d82a\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " Oct 30 15:44:34.548878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548699 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-kubeconfig\") pod \"520d728b-ffb8-4845-94f2-00ca8968d82a\" (UID: \"520d728b-ffb8-4845-94f2-00ca8968d82a\") " Oct 30 15:44:34.548992 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.548956 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-ca-configmap" pod "520d728b-ffb8-4845-94f2-00ca8968d82a" (UID: "520d728b-ffb8-4845-94f2-00ca8968d82a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:44:34.550910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.550884 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-token" pod "520d728b-ffb8-4845-94f2-00ca8968d82a" (UID: "520d728b-ffb8-4845-94f2-00ca8968d82a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:44:34.551019 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.550919 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-kubeconfig" pod "520d728b-ffb8-4845-94f2-00ca8968d82a" (UID: "520d728b-ffb8-4845-94f2-00ca8968d82a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:44:34.551154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.551128 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520d728b-ffb8-4845-94f2-00ca8968d82a-local-certs" pod "520d728b-ffb8-4845-94f2-00ca8968d82a" (UID: "520d728b-ffb8-4845-94f2-00ca8968d82a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:44:34.551239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.551152 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-kube-api-access-fhshv" pod "520d728b-ffb8-4845-94f2-00ca8968d82a" (UID: "520d728b-ffb8-4845-94f2-00ca8968d82a"). InnerVolumeSpecName "kube-api-access-fhshv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:44:34.551239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.551179 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-dns-cert" pod "520d728b-ffb8-4845-94f2-00ca8968d82a" (UID: "520d728b-ffb8-4845-94f2-00ca8968d82a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:44:34.551704 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.551687 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-cacerts" pod "520d728b-ffb8-4845-94f2-00ca8968d82a" (UID: "520d728b-ffb8-4845-94f2-00ca8968d82a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:44:34.649417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.649333 4401 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-kubeconfig\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:44:34.649417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.649363 4401 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-dns-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:44:34.649417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.649378 4401 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-csr-ca-configmap\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:44:34.649417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.649391 4401 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/520d728b-ffb8-4845-94f2-00ca8968d82a-cacerts\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:44:34.649417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.649404 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhshv\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-kube-api-access-fhshv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:44:34.649417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.649418 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/520d728b-ffb8-4845-94f2-00ca8968d82a-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:44:34.649733 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:34.649430 4401 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/520d728b-ffb8-4845-94f2-00ca8968d82a-local-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:44:35.101028 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.100999 4401 generic.go:356] "Generic (PLEG): container finished" podID="520d728b-ffb8-4845-94f2-00ca8968d82a" containerID="dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53" exitCode=0 Oct 30 15:44:35.101552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.101059 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" Oct 30 15:44:35.101552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.101091 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" event={"ID":"520d728b-ffb8-4845-94f2-00ca8968d82a","Type":"ContainerDied","Data":"dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53"} Oct 30 15:44:35.101552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.101153 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5" event={"ID":"520d728b-ffb8-4845-94f2-00ca8968d82a","Type":"ContainerDied","Data":"c71542ad0e5e1945e0fc9458ee7b7eba32981f9fe7b5ce63e207ad53f86603d4"} Oct 30 15:44:35.101552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.101177 4401 scope.go:122] "RemoveContainer" containerID="dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53" Oct 30 15:44:35.110030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.110013 4401 scope.go:122] "RemoveContainer" containerID="dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53" Oct 30 15:44:35.110297 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:44:35.110281 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53\": container with ID starting with dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53 not found: ID does not exist" containerID="dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53" Oct 30 15:44:35.110370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.110316 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53"} err="failed to get container status \"dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53\": rpc error: code = NotFound desc = could not find container \"dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53\": container with ID starting with dd7c2cd576621ed73a9ce9e261dde6f4353981475494288a7e39960dc6f94e53 not found: ID does not exist" Oct 30 15:44:35.185592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.185562 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5"] Oct 30 15:44:35.199719 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.199684 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mdlr5"] Oct 30 15:44:35.426097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:44:35.426023 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520d728b-ffb8-4845-94f2-00ca8968d82a" path="/var/lib/kubelet/pods/520d728b-ffb8-4845-94f2-00ca8968d82a/volumes" Oct 30 15:44:51.965967 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:44:51.965930 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:45:21.965037 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:45:21.965000 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:45:23.279330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.279298 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s"] Oct 30 15:45:23.279745 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.279671 4401 cpu_manager.go:414] "containerMap: removing container" podUID="520d728b-ffb8-4845-94f2-00ca8968d82a" containerName="discovery" Oct 30 15:45:23.279745 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.279682 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520d728b-ffb8-4845-94f2-00ca8968d82a" containerName="discovery" Oct 30 15:45:23.279846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.279755 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="520d728b-ffb8-4845-94f2-00ca8968d82a" containerName="discovery" Oct 30 15:45:23.284292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.284273 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.302904 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.302880 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s"] Oct 30 15:45:23.314586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.314561 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 15:45:23.314695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.314592 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 15:45:23.314791 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.314774 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-n86p8\"" Oct 30 15:45:23.357710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.357683 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv"] Oct 30 15:45:23.361475 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.361459 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.379294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.379269 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv"] Oct 30 15:45:23.386032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.386015 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-t4wz8\"" Oct 30 15:45:23.447066 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.447045 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.447188 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.447096 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.447188 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.447171 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdzqq\" (UniqueName: \"kubernetes.io/projected/00550232-4397-44a8-9e4c-50c15328d7ba-kube-api-access-fdzqq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.447351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.447203 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.447351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.447220 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.447351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.447305 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqxrf\" (UniqueName: \"kubernetes.io/projected/7664a97c-34bb-42f7-ae7f-5402d7a08380-kube-api-access-sqxrf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.491375 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.491351 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn"] Oct 30 15:45:23.495035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.495018 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.512527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.512507 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-wtg5t\"" Oct 30 15:45:23.514066 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.514046 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn"] Oct 30 15:45:23.548140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548080 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.548140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548131 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.548307 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548164 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqxrf\" (UniqueName: \"kubernetes.io/projected/7664a97c-34bb-42f7-ae7f-5402d7a08380-kube-api-access-sqxrf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.548307 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548209 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.548307 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548278 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.548448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548356 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdzqq\" (UniqueName: \"kubernetes.io/projected/00550232-4397-44a8-9e4c-50c15328d7ba-kube-api-access-fdzqq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.548591 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548571 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.548958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.548940 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.549081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.549063 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.549120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.549068 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.562865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.562843 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k"] Oct 30 15:45:23.566495 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.566479 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.574647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.574623 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdzqq\" (UniqueName: \"kubernetes.io/projected/00550232-4397-44a8-9e4c-50c15328d7ba-kube-api-access-fdzqq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.574647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.574635 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqxrf\" (UniqueName: \"kubernetes.io/projected/7664a97c-34bb-42f7-ae7f-5402d7a08380-kube-api-access-sqxrf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.585951 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.585931 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k"] Oct 30 15:45:23.600631 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.600614 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:23.612399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.612377 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-p4ffx\"" Oct 30 15:45:23.649080 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.648954 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bznqv\" (UniqueName: \"kubernetes.io/projected/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-kube-api-access-bznqv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.649080 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.649052 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.649216 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.649091 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.677737 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.677711 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:23.750439 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750413 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.750562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750475 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bznqv\" (UniqueName: \"kubernetes.io/projected/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-kube-api-access-bznqv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.750562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750503 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdntl\" (UniqueName: \"kubernetes.io/projected/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-kube-api-access-xdntl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.750562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750541 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.750562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750559 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.750764 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750685 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.750869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750844 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.750934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.750918 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.775757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.775731 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bznqv\" (UniqueName: \"kubernetes.io/projected/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-kube-api-access-bznqv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.811738 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.811691 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:23.852158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.852126 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdntl\" (UniqueName: \"kubernetes.io/projected/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-kube-api-access-xdntl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.852402 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.852211 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.852402 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.852244 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.852627 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.852593 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.852761 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.852660 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.878539 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.878504 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdntl\" (UniqueName: \"kubernetes.io/projected/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-kube-api-access-xdntl\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.883229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.883211 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:23.902904 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.902878 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s"] Oct 30 15:45:23.904497 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:45:23.904473 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00550232_4397_44a8_9e4c_50c15328d7ba.slice/crio-70abfa851f4f7b07124e66699b969654b4895746ff824beff43af2919a74b945 WatchSource:0}: Error finding container 70abfa851f4f7b07124e66699b969654b4895746ff824beff43af2919a74b945: Status 404 returned error can't find the container with id 70abfa851f4f7b07124e66699b969654b4895746ff824beff43af2919a74b945 Oct 30 15:45:23.983541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:23.983512 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv"] Oct 30 15:45:23.985515 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:45:23.985491 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7664a97c_34bb_42f7_ae7f_5402d7a08380.slice/crio-cc6c0a132c81c3e013a57f1add53f36ccafa608a6b23aee6d26b44e37c0448f1 WatchSource:0}: Error finding container cc6c0a132c81c3e013a57f1add53f36ccafa608a6b23aee6d26b44e37c0448f1: Status 404 returned error can't find the container with id cc6c0a132c81c3e013a57f1add53f36ccafa608a6b23aee6d26b44e37c0448f1 Oct 30 15:45:24.115067 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.115035 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn"] Oct 30 15:45:24.116209 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:45:24.116174 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71d3a813_e8e5_4d5b_a6fe_86d8b00fcf69.slice/crio-c586fbc9e6be47975ea5c7ee47d9ae15e6ff52519b274ec02160ac03601f597d WatchSource:0}: Error finding container c586fbc9e6be47975ea5c7ee47d9ae15e6ff52519b274ec02160ac03601f597d: Status 404 returned error can't find the container with id c586fbc9e6be47975ea5c7ee47d9ae15e6ff52519b274ec02160ac03601f597d Oct 30 15:45:24.189476 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.189450 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k"] Oct 30 15:45:24.222137 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:45:24.222106 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd252a3_5e29_4718_91c5_eb06ac4dae2b.slice/crio-e5097d233b380147d5f1864f5f8b0cbc21aefa46f53f99e50c0686a94c1b35f2 WatchSource:0}: Error finding container e5097d233b380147d5f1864f5f8b0cbc21aefa46f53f99e50c0686a94c1b35f2: Status 404 returned error can't find the container with id e5097d233b380147d5f1864f5f8b0cbc21aefa46f53f99e50c0686a94c1b35f2 Oct 30 15:45:24.323156 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.323098 4401 generic.go:356] "Generic (PLEG): container finished" podID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerID="b5112aeeb34b600f869fd15b2daa3a22b802d40b7b07d231c9715373b9592d3c" exitCode=0 Oct 30 15:45:24.323519 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.323154 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" event={"ID":"7fd252a3-5e29-4718-91c5-eb06ac4dae2b","Type":"ContainerDied","Data":"b5112aeeb34b600f869fd15b2daa3a22b802d40b7b07d231c9715373b9592d3c"} Oct 30 15:45:24.323519 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.323205 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" event={"ID":"7fd252a3-5e29-4718-91c5-eb06ac4dae2b","Type":"ContainerStarted","Data":"e5097d233b380147d5f1864f5f8b0cbc21aefa46f53f99e50c0686a94c1b35f2"} Oct 30 15:45:24.324775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.324749 4401 generic.go:356] "Generic (PLEG): container finished" podID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerID="0bd3e485daaeb4f0dd218d738400e93a1738ab9b83193dca75f859f2980fbf75" exitCode=0 Oct 30 15:45:24.324897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.324779 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" event={"ID":"7664a97c-34bb-42f7-ae7f-5402d7a08380","Type":"ContainerDied","Data":"0bd3e485daaeb4f0dd218d738400e93a1738ab9b83193dca75f859f2980fbf75"} Oct 30 15:45:24.324897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.324814 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" event={"ID":"7664a97c-34bb-42f7-ae7f-5402d7a08380","Type":"ContainerStarted","Data":"cc6c0a132c81c3e013a57f1add53f36ccafa608a6b23aee6d26b44e37c0448f1"} Oct 30 15:45:24.326629 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.326525 4401 generic.go:356] "Generic (PLEG): container finished" podID="00550232-4397-44a8-9e4c-50c15328d7ba" containerID="0493a2649adfef12cbf16f2b556b0ba7b458ef31557b160beb09fc403dcd8665" exitCode=0 Oct 30 15:45:24.326629 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.326605 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" event={"ID":"00550232-4397-44a8-9e4c-50c15328d7ba","Type":"ContainerDied","Data":"0493a2649adfef12cbf16f2b556b0ba7b458ef31557b160beb09fc403dcd8665"} Oct 30 15:45:24.326729 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.326649 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" event={"ID":"00550232-4397-44a8-9e4c-50c15328d7ba","Type":"ContainerStarted","Data":"70abfa851f4f7b07124e66699b969654b4895746ff824beff43af2919a74b945"} Oct 30 15:45:24.328315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.328173 4401 generic.go:356] "Generic (PLEG): container finished" podID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerID="1376a912fdb1792b6264835ab8d0f959bbe8d17f5b8a3ce40dc6e2123c5acc9a" exitCode=0 Oct 30 15:45:24.328315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.328222 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" event={"ID":"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69","Type":"ContainerDied","Data":"1376a912fdb1792b6264835ab8d0f959bbe8d17f5b8a3ce40dc6e2123c5acc9a"} Oct 30 15:45:24.328315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:24.328258 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" event={"ID":"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69","Type":"ContainerStarted","Data":"c586fbc9e6be47975ea5c7ee47d9ae15e6ff52519b274ec02160ac03601f597d"} Oct 30 15:45:25.334997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.334917 4401 generic.go:356] "Generic (PLEG): container finished" podID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerID="9bde9228264be8c9ea280fdacc447e92eb29e84acedad0a9c20181e0064eb3ae" exitCode=0 Oct 30 15:45:25.335422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.334996 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" event={"ID":"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69","Type":"ContainerDied","Data":"9bde9228264be8c9ea280fdacc447e92eb29e84acedad0a9c20181e0064eb3ae"} Oct 30 15:45:25.336936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.336915 4401 generic.go:356] "Generic (PLEG): container finished" podID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerID="3609fedac6e417b0f216181a02f09375ca6b2b0373c86c8f64fd01603b150d6f" exitCode=0 Oct 30 15:45:25.337029 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.336993 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" event={"ID":"7fd252a3-5e29-4718-91c5-eb06ac4dae2b","Type":"ContainerDied","Data":"3609fedac6e417b0f216181a02f09375ca6b2b0373c86c8f64fd01603b150d6f"} Oct 30 15:45:25.339020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.338996 4401 generic.go:356] "Generic (PLEG): container finished" podID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerID="9914b8eecb8d0a45f375ff353eda4a3679a6106420137df6692df0ef2a9e4f17" exitCode=0 Oct 30 15:45:25.339128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.339072 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" event={"ID":"7664a97c-34bb-42f7-ae7f-5402d7a08380","Type":"ContainerDied","Data":"9914b8eecb8d0a45f375ff353eda4a3679a6106420137df6692df0ef2a9e4f17"} Oct 30 15:45:25.340837 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.340802 4401 generic.go:356] "Generic (PLEG): container finished" podID="00550232-4397-44a8-9e4c-50c15328d7ba" containerID="c2ceb0b6d63a537431b4bf26b46b024224b7a891087ba7e1baeda7fe22e416f0" exitCode=0 Oct 30 15:45:25.340902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:25.340853 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" event={"ID":"00550232-4397-44a8-9e4c-50c15328d7ba","Type":"ContainerDied","Data":"c2ceb0b6d63a537431b4bf26b46b024224b7a891087ba7e1baeda7fe22e416f0"} Oct 30 15:45:26.347846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.347799 4401 generic.go:356] "Generic (PLEG): container finished" podID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerID="4715a80403a76c0f32d93b722df4eac3b07a98453e2a01b1c5ecce4e13382d6d" exitCode=0 Oct 30 15:45:26.348227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.347881 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" event={"ID":"7fd252a3-5e29-4718-91c5-eb06ac4dae2b","Type":"ContainerDied","Data":"4715a80403a76c0f32d93b722df4eac3b07a98453e2a01b1c5ecce4e13382d6d"} Oct 30 15:45:26.349963 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.349946 4401 generic.go:356] "Generic (PLEG): container finished" podID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerID="8e1e049bf5d9318f44a60d726df5dea283ad70b7e8f510a0b73b8bde4f3c0677" exitCode=0 Oct 30 15:45:26.350053 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.350005 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" event={"ID":"7664a97c-34bb-42f7-ae7f-5402d7a08380","Type":"ContainerDied","Data":"8e1e049bf5d9318f44a60d726df5dea283ad70b7e8f510a0b73b8bde4f3c0677"} Oct 30 15:45:26.351796 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.351774 4401 generic.go:356] "Generic (PLEG): container finished" podID="00550232-4397-44a8-9e4c-50c15328d7ba" containerID="5f1f3de551856e730848cff0b044c2ab71dcec077ae372e7517f267d29dfedb6" exitCode=0 Oct 30 15:45:26.351903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.351850 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" event={"ID":"00550232-4397-44a8-9e4c-50c15328d7ba","Type":"ContainerDied","Data":"5f1f3de551856e730848cff0b044c2ab71dcec077ae372e7517f267d29dfedb6"} Oct 30 15:45:26.353738 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.353719 4401 generic.go:356] "Generic (PLEG): container finished" podID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerID="be189bd32c26be7f09e6b90be4a6c608e27fdb20969d4611a28d961ec795cb22" exitCode=0 Oct 30 15:45:26.353819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:26.353757 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" event={"ID":"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69","Type":"ContainerDied","Data":"be189bd32c26be7f09e6b90be4a6c608e27fdb20969d4611a28d961ec795cb22"} Oct 30 15:45:27.522158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.522132 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:27.576811 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.576788 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:27.579993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.579970 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:27.583612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.583594 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:27.687336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687265 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-bundle\") pod \"7664a97c-34bb-42f7-ae7f-5402d7a08380\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " Oct 30 15:45:27.687336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687301 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-kube-api-access-bznqv\" (UniqueName: \"kubernetes.io/projected/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-kube-api-access-bznqv\") pod \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " Oct 30 15:45:27.687336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687322 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-bundle\") pod \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " Oct 30 15:45:27.687603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687377 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-bundle\") pod \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " Oct 30 15:45:27.687603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687424 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-util\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-util\") pod \"7664a97c-34bb-42f7-ae7f-5402d7a08380\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " Oct 30 15:45:27.687603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687467 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7664a97c-34bb-42f7-ae7f-5402d7a08380-kube-api-access-sqxrf\" (UniqueName: \"kubernetes.io/projected/7664a97c-34bb-42f7-ae7f-5402d7a08380-kube-api-access-sqxrf\") pod \"7664a97c-34bb-42f7-ae7f-5402d7a08380\" (UID: \"7664a97c-34bb-42f7-ae7f-5402d7a08380\") " Oct 30 15:45:27.687603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687531 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-util\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-util\") pod \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\" (UID: \"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69\") " Oct 30 15:45:27.687603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687565 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-bundle\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-bundle\") pod \"00550232-4397-44a8-9e4c-50c15328d7ba\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " Oct 30 15:45:27.687869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687604 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-util\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-util\") pod \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " Oct 30 15:45:27.687869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687637 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00550232-4397-44a8-9e4c-50c15328d7ba-kube-api-access-fdzqq\" (UniqueName: \"kubernetes.io/projected/00550232-4397-44a8-9e4c-50c15328d7ba-kube-api-access-fdzqq\") pod \"00550232-4397-44a8-9e4c-50c15328d7ba\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " Oct 30 15:45:27.687869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687675 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-util\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-util\") pod \"00550232-4397-44a8-9e4c-50c15328d7ba\" (UID: \"00550232-4397-44a8-9e4c-50c15328d7ba\") " Oct 30 15:45:27.687869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687733 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-kube-api-access-xdntl\" (UniqueName: \"kubernetes.io/projected/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-kube-api-access-xdntl\") pod \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\" (UID: \"7fd252a3-5e29-4718-91c5-eb06ac4dae2b\") " Oct 30 15:45:27.688050 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.687990 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-bundle" pod "7fd252a3-5e29-4718-91c5-eb06ac4dae2b" (UID: "7fd252a3-5e29-4718-91c5-eb06ac4dae2b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.688337 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.688167 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-bundle" pod "71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" (UID: "71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.688337 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.688283 4401 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.688337 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.688307 4401 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.688606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.688400 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-bundle" pod "7664a97c-34bb-42f7-ae7f-5402d7a08380" (UID: "7664a97c-34bb-42f7-ae7f-5402d7a08380"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.688865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.688814 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-bundle" pod "00550232-4397-44a8-9e4c-50c15328d7ba" (UID: "00550232-4397-44a8-9e4c-50c15328d7ba"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.690614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.690587 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-kube-api-access-bznqv" pod "71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" (UID: "71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69"). InnerVolumeSpecName "kube-api-access-bznqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:27.690614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.690602 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-kube-api-access-xdntl" pod "7fd252a3-5e29-4718-91c5-eb06ac4dae2b" (UID: "7fd252a3-5e29-4718-91c5-eb06ac4dae2b"). InnerVolumeSpecName "kube-api-access-xdntl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:27.690722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.690676 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7664a97c-34bb-42f7-ae7f-5402d7a08380-kube-api-access-sqxrf" pod "7664a97c-34bb-42f7-ae7f-5402d7a08380" (UID: "7664a97c-34bb-42f7-ae7f-5402d7a08380"). InnerVolumeSpecName "kube-api-access-sqxrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:27.694543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.694505 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-util" pod "00550232-4397-44a8-9e4c-50c15328d7ba" (UID: "00550232-4397-44a8-9e4c-50c15328d7ba"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.694647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.694628 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-util" pod "71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" (UID: "71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.694812 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.694798 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00550232-4397-44a8-9e4c-50c15328d7ba-kube-api-access-fdzqq" pod "00550232-4397-44a8-9e4c-50c15328d7ba" (UID: "00550232-4397-44a8-9e4c-50c15328d7ba"). InnerVolumeSpecName "kube-api-access-fdzqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:27.695756 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.695731 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-util" pod "7fd252a3-5e29-4718-91c5-eb06ac4dae2b" (UID: "7fd252a3-5e29-4718-91c5-eb06ac4dae2b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.697492 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.697473 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-util" pod "7664a97c-34bb-42f7-ae7f-5402d7a08380" (UID: "7664a97c-34bb-42f7-ae7f-5402d7a08380"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:45:27.789386 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789357 4401 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-util\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789386 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789386 4401 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789401 4401 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-util\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789414 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdzqq\" (UniqueName: \"kubernetes.io/projected/00550232-4397-44a8-9e4c-50c15328d7ba-kube-api-access-fdzqq\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789428 4401 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/00550232-4397-44a8-9e4c-50c15328d7ba-util\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789440 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdntl\" (UniqueName: \"kubernetes.io/projected/7fd252a3-5e29-4718-91c5-eb06ac4dae2b-kube-api-access-xdntl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789452 4401 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-bundle\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789464 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bznqv\" (UniqueName: \"kubernetes.io/projected/71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69-kube-api-access-bznqv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789476 4401 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7664a97c-34bb-42f7-ae7f-5402d7a08380-util\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:27.789547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:27.789488 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqxrf\" (UniqueName: \"kubernetes.io/projected/7664a97c-34bb-42f7-ae7f-5402d7a08380-kube-api-access-sqxrf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:45:28.366276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.366248 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" Oct 30 15:45:28.366457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.366246 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmwq9k" event={"ID":"7fd252a3-5e29-4718-91c5-eb06ac4dae2b","Type":"ContainerDied","Data":"e5097d233b380147d5f1864f5f8b0cbc21aefa46f53f99e50c0686a94c1b35f2"} Oct 30 15:45:28.366457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.366384 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5097d233b380147d5f1864f5f8b0cbc21aefa46f53f99e50c0686a94c1b35f2" Oct 30 15:45:28.368254 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.368221 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" event={"ID":"7664a97c-34bb-42f7-ae7f-5402d7a08380","Type":"ContainerDied","Data":"cc6c0a132c81c3e013a57f1add53f36ccafa608a6b23aee6d26b44e37c0448f1"} Oct 30 15:45:28.368379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.368261 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc6c0a132c81c3e013a57f1add53f36ccafa608a6b23aee6d26b44e37c0448f1" Oct 30 15:45:28.368379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.368231 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lwsfv" Oct 30 15:45:28.370249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.370226 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" Oct 30 15:45:28.370377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.370249 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7x2x9s" event={"ID":"00550232-4397-44a8-9e4c-50c15328d7ba","Type":"ContainerDied","Data":"70abfa851f4f7b07124e66699b969654b4895746ff824beff43af2919a74b945"} Oct 30 15:45:28.370377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.370289 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70abfa851f4f7b07124e66699b969654b4895746ff824beff43af2919a74b945" Oct 30 15:45:28.372322 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.372295 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" event={"ID":"71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69","Type":"ContainerDied","Data":"c586fbc9e6be47975ea5c7ee47d9ae15e6ff52519b274ec02160ac03601f597d"} Oct 30 15:45:28.372322 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.372321 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afrrzn" Oct 30 15:45:28.372505 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:28.372338 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c586fbc9e6be47975ea5c7ee47d9ae15e6ff52519b274ec02160ac03601f597d" Oct 30 15:45:34.813372 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813337 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-2rdzt"] Oct 30 15:45:34.813789 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813772 4401 cpu_manager.go:414] "containerMap: removing container" podUID="00550232-4397-44a8-9e4c-50c15328d7ba" containerName="extract" Oct 30 15:45:34.813858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813791 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00550232-4397-44a8-9e4c-50c15328d7ba" containerName="extract" Oct 30 15:45:34.813858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813803 4401 cpu_manager.go:414] "containerMap: removing container" podUID="00550232-4397-44a8-9e4c-50c15328d7ba" containerName="pull" Oct 30 15:45:34.813858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813808 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00550232-4397-44a8-9e4c-50c15328d7ba" containerName="pull" Oct 30 15:45:34.813858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813815 4401 cpu_manager.go:414] "containerMap: removing container" podUID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerName="pull" Oct 30 15:45:34.813858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813820 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerName="pull" Oct 30 15:45:34.813858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813855 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerName="pull" Oct 30 15:45:34.813858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813859 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerName="pull" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813873 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813878 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813885 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813889 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813916 4401 cpu_manager.go:414] "containerMap: removing container" podUID="00550232-4397-44a8-9e4c-50c15328d7ba" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813922 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00550232-4397-44a8-9e4c-50c15328d7ba" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813927 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813932 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813942 4401 cpu_manager.go:414] "containerMap: removing container" podUID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813947 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerName="util" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813953 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerName="pull" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813957 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerName="pull" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813967 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813971 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813977 4401 cpu_manager.go:414] "containerMap: removing container" podUID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.813982 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.814038 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7664a97c-34bb-42f7-ae7f-5402d7a08380" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.814046 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd252a3-5e29-4718-91c5-eb06ac4dae2b" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.814052 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="71d3a813-e8e5-4d5b-a6fe-86d8b00fcf69" containerName="extract" Oct 30 15:45:34.814061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.814059 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="00550232-4397-44a8-9e4c-50c15328d7ba" containerName="extract" Oct 30 15:45:34.818505 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.818488 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" Oct 30 15:45:34.837675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.837641 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gzgt6\"" Oct 30 15:45:34.841072 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.841047 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-2rdzt"] Oct 30 15:45:34.850600 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.850573 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8f2\" (UniqueName: \"kubernetes.io/projected/f4102670-ceac-42d4-8a3c-b685f5d4d43c-kube-api-access-gr8f2\") pod \"authorino-operator-68d6dd76dc-2rdzt\" (UID: \"f4102670-ceac-42d4-8a3c-b685f5d4d43c\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" Oct 30 15:45:34.851595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.851575 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 15:45:34.851655 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.851597 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 15:45:34.951977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.951939 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8f2\" (UniqueName: \"kubernetes.io/projected/f4102670-ceac-42d4-8a3c-b685f5d4d43c-kube-api-access-gr8f2\") pod \"authorino-operator-68d6dd76dc-2rdzt\" (UID: \"f4102670-ceac-42d4-8a3c-b685f5d4d43c\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" Oct 30 15:45:34.980344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:34.980313 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8f2\" (UniqueName: \"kubernetes.io/projected/f4102670-ceac-42d4-8a3c-b685f5d4d43c-kube-api-access-gr8f2\") pod \"authorino-operator-68d6dd76dc-2rdzt\" (UID: \"f4102670-ceac-42d4-8a3c-b685f5d4d43c\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" Oct 30 15:45:35.130692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:35.130663 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" Oct 30 15:45:35.452481 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:35.452455 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-2rdzt"] Oct 30 15:45:35.454491 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:45:35.454464 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4102670_ceac_42d4_8a3c_b685f5d4d43c.slice/crio-264cacc9fa1b2820e03db4404f37ff0fafd6fc03e981ea85db3e757405b7b26d WatchSource:0}: Error finding container 264cacc9fa1b2820e03db4404f37ff0fafd6fc03e981ea85db3e757405b7b26d: Status 404 returned error can't find the container with id 264cacc9fa1b2820e03db4404f37ff0fafd6fc03e981ea85db3e757405b7b26d Oct 30 15:45:36.416631 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:36.416588 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" event={"ID":"f4102670-ceac-42d4-8a3c-b685f5d4d43c","Type":"ContainerStarted","Data":"264cacc9fa1b2820e03db4404f37ff0fafd6fc03e981ea85db3e757405b7b26d"} Oct 30 15:45:37.132055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.132020 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx"] Oct 30 15:45:37.142679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.142655 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:45:37.159861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.159812 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx"] Oct 30 15:45:37.171041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.171014 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-sv2lf\"" Oct 30 15:45:37.172215 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.172190 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whjkb\" (UniqueName: \"kubernetes.io/projected/3d6aba08-1940-4e35-8fc9-f1569e292f3a-kube-api-access-whjkb\") pod \"limitador-operator-controller-manager-856dd584c5-24nvx\" (UID: \"3d6aba08-1940-4e35-8fc9-f1569e292f3a\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:45:37.273302 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.273272 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whjkb\" (UniqueName: \"kubernetes.io/projected/3d6aba08-1940-4e35-8fc9-f1569e292f3a-kube-api-access-whjkb\") pod \"limitador-operator-controller-manager-856dd584c5-24nvx\" (UID: \"3d6aba08-1940-4e35-8fc9-f1569e292f3a\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:45:37.298548 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.298517 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whjkb\" (UniqueName: \"kubernetes.io/projected/3d6aba08-1940-4e35-8fc9-f1569e292f3a-kube-api-access-whjkb\") pod \"limitador-operator-controller-manager-856dd584c5-24nvx\" (UID: \"3d6aba08-1940-4e35-8fc9-f1569e292f3a\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:45:37.459026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.458949 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:45:37.773319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:37.773293 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx"] Oct 30 15:45:37.774490 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:45:37.774462 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6aba08_1940_4e35_8fc9_f1569e292f3a.slice/crio-50a6c7ea050226590d974ca817c4542257341a181d59bda40df7a7271d2d2786 WatchSource:0}: Error finding container 50a6c7ea050226590d974ca817c4542257341a181d59bda40df7a7271d2d2786: Status 404 returned error can't find the container with id 50a6c7ea050226590d974ca817c4542257341a181d59bda40df7a7271d2d2786 Oct 30 15:45:38.430595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:38.430562 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" event={"ID":"f4102670-ceac-42d4-8a3c-b685f5d4d43c","Type":"ContainerStarted","Data":"a6885f97f5c74d7e3d566baf6dd63835c8efe530be992e95809c773ac10ccf9e"} Oct 30 15:45:38.430778 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:38.430657 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" Oct 30 15:45:38.432307 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:38.432284 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" event={"ID":"3d6aba08-1940-4e35-8fc9-f1569e292f3a","Type":"ContainerStarted","Data":"50a6c7ea050226590d974ca817c4542257341a181d59bda40df7a7271d2d2786"} Oct 30 15:45:38.481504 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:38.481460 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" podStartSLOduration=2.329502198 podStartE2EDuration="4.481446917s" totalImagesPullingTime="2.151944719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:45:38.479068795 +0000 UTC m=+447.178341952" watchObservedRunningTime="2026-10-30 15:45:38.481446917 +0000 UTC m=+447.180720073" Oct 30 15:45:39.438031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:39.437942 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" event={"ID":"3d6aba08-1940-4e35-8fc9-f1569e292f3a","Type":"ContainerStarted","Data":"e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e"} Oct 30 15:45:39.438272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:39.438259 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:45:39.487678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:39.487628 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" podStartSLOduration=1.160277818 podStartE2EDuration="2.487615641s" totalImagesPullingTime="1.327337823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:45:39.48532569 +0000 UTC m=+448.184598844" watchObservedRunningTime="2026-10-30 15:45:39.487615641 +0000 UTC m=+448.186888797" Oct 30 15:45:49.440032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:49.439997 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-2rdzt" Oct 30 15:45:50.444507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:45:50.444473 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:45:51.968965 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:45:51.968928 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:46:00.909007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:00.908975 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb"] Oct 30 15:46:00.913777 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:00.913754 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:00.924208 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:00.924184 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb"] Oct 30 15:46:00.935793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:00.935771 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-trpmg\"" Oct 30 15:46:00.986547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:00.986512 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8k46\" (UniqueName: \"kubernetes.io/projected/8f113622-3c9c-44ef-8079-7abaa0850d4e-kube-api-access-x8k46\") pod \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:00.986713 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:00.986573 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f113622-3c9c-44ef-8079-7abaa0850d4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:01.087999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:01.087967 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f113622-3c9c-44ef-8079-7abaa0850d4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:01.088160 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:01.088047 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8k46\" (UniqueName: \"kubernetes.io/projected/8f113622-3c9c-44ef-8079-7abaa0850d4e-kube-api-access-x8k46\") pod \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:01.088403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:01.088385 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f113622-3c9c-44ef-8079-7abaa0850d4e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:01.130389 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:01.130357 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8k46\" (UniqueName: \"kubernetes.io/projected/8f113622-3c9c-44ef-8079-7abaa0850d4e-kube-api-access-x8k46\") pod \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:01.227007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:01.226917 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:01.557983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:01.557910 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb"] Oct 30 15:46:01.574179 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:46:01.574146 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f113622_3c9c_44ef_8079_7abaa0850d4e.slice/crio-369f17a300af83fa61577d9f4292120c1d3820ea95a555ccb5c1167ddf0d4183 WatchSource:0}: Error finding container 369f17a300af83fa61577d9f4292120c1d3820ea95a555ccb5c1167ddf0d4183: Status 404 returned error can't find the container with id 369f17a300af83fa61577d9f4292120c1d3820ea95a555ccb5c1167ddf0d4183 Oct 30 15:46:02.556138 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:02.556101 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" event={"ID":"8f113622-3c9c-44ef-8079-7abaa0850d4e","Type":"ContainerStarted","Data":"369f17a300af83fa61577d9f4292120c1d3820ea95a555ccb5c1167ddf0d4183"} Oct 30 15:46:03.217782 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.217752 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb"] Oct 30 15:46:03.220424 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.220396 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb"] Oct 30 15:46:03.233594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.233568 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl"] Oct 30 15:46:03.237346 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.237324 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.241638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.241614 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx"] Oct 30 15:46:03.241850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.241811 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" podUID="3d6aba08-1940-4e35-8fc9-f1569e292f3a" containerName="manager" containerID="cri-o://e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e" gracePeriod=2 Oct 30 15:46:03.245875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.245851 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx"] Oct 30 15:46:03.267932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.267906 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm"] Oct 30 15:46:03.268469 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.268446 4401 cpu_manager.go:414] "containerMap: removing container" podUID="3d6aba08-1940-4e35-8fc9-f1569e292f3a" containerName="manager" Oct 30 15:46:03.268469 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.268469 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6aba08-1940-4e35-8fc9-f1569e292f3a" containerName="manager" Oct 30 15:46:03.268631 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.268574 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6aba08-1940-4e35-8fc9-f1569e292f3a" containerName="manager" Oct 30 15:46:03.273553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.273533 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl"] Oct 30 15:46:03.273662 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.273626 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" Oct 30 15:46:03.293600 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.293524 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm"] Oct 30 15:46:03.307283 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.307236 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddf09cc-81ce-4d72-9564-b1ed0704e183-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-8tqpl\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.307458 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.307351 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmmm9\" (UniqueName: \"kubernetes.io/projected/1ddf09cc-81ce-4d72-9564-b1ed0704e183-kube-api-access-dmmm9\") pod \"kuadrant-operator-controller-manager-6b76784d48-8tqpl\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.370694 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:03.370657 4401 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-24nvx\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="3d6aba08-1940-4e35-8fc9-f1569e292f3a" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:46:03.408231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.408199 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddf09cc-81ce-4d72-9564-b1ed0704e183-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-8tqpl\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.408375 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.408256 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rvv\" (UniqueName: \"kubernetes.io/projected/1b0b1350-4d9c-4012-8761-ddccf2e06879-kube-api-access-r9rvv\") pod \"limitador-operator-controller-manager-856dd584c5-dx5wm\" (UID: \"1b0b1350-4d9c-4012-8761-ddccf2e06879\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" Oct 30 15:46:03.408375 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.408296 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmmm9\" (UniqueName: \"kubernetes.io/projected/1ddf09cc-81ce-4d72-9564-b1ed0704e183-kube-api-access-dmmm9\") pod \"kuadrant-operator-controller-manager-6b76784d48-8tqpl\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.408808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.408785 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddf09cc-81ce-4d72-9564-b1ed0704e183-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-8tqpl\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.438737 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.438706 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmmm9\" (UniqueName: \"kubernetes.io/projected/1ddf09cc-81ce-4d72-9564-b1ed0704e183-kube-api-access-dmmm9\") pod \"kuadrant-operator-controller-manager-6b76784d48-8tqpl\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.509299 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.509271 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rvv\" (UniqueName: \"kubernetes.io/projected/1b0b1350-4d9c-4012-8761-ddccf2e06879-kube-api-access-r9rvv\") pod \"limitador-operator-controller-manager-856dd584c5-dx5wm\" (UID: \"1b0b1350-4d9c-4012-8761-ddccf2e06879\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" Oct 30 15:46:03.522774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.522747 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:46:03.537284 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.537257 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rvv\" (UniqueName: \"kubernetes.io/projected/1b0b1350-4d9c-4012-8761-ddccf2e06879-kube-api-access-r9rvv\") pod \"limitador-operator-controller-manager-856dd584c5-dx5wm\" (UID: \"1b0b1350-4d9c-4012-8761-ddccf2e06879\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" Oct 30 15:46:03.555327 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.555301 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:03.563307 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.563277 4401 generic.go:356] "Generic (PLEG): container finished" podID="3d6aba08-1940-4e35-8fc9-f1569e292f3a" containerID="e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e" exitCode=0 Oct 30 15:46:03.563693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.563334 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-24nvx" Oct 30 15:46:03.563693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.563429 4401 scope.go:122] "RemoveContainer" containerID="e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e" Oct 30 15:46:03.574219 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.574179 4401 scope.go:122] "RemoveContainer" containerID="e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e" Oct 30 15:46:03.574581 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:03.574557 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e\": container with ID starting with e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e not found: ID does not exist" containerID="e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e" Oct 30 15:46:03.574669 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.574593 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e"} err="failed to get container status \"e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e\": rpc error: code = NotFound desc = could not find container \"e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e\": container with ID starting with e3a313adabf7b89e6e0cf1dc3984ffbdfde1eafed75dbd45a1dc9ddaaebd994e not found: ID does not exist" Oct 30 15:46:03.611087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.610205 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6aba08-1940-4e35-8fc9-f1569e292f3a-kube-api-access-whjkb\" (UniqueName: \"kubernetes.io/projected/3d6aba08-1940-4e35-8fc9-f1569e292f3a-kube-api-access-whjkb\") pod \"3d6aba08-1940-4e35-8fc9-f1569e292f3a\" (UID: \"3d6aba08-1940-4e35-8fc9-f1569e292f3a\") " Oct 30 15:46:03.613440 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.612862 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6aba08-1940-4e35-8fc9-f1569e292f3a-kube-api-access-whjkb" pod "3d6aba08-1940-4e35-8fc9-f1569e292f3a" (UID: "3d6aba08-1940-4e35-8fc9-f1569e292f3a"). InnerVolumeSpecName "kube-api-access-whjkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:46:03.616233 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.615974 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" Oct 30 15:46:03.711623 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.711594 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whjkb\" (UniqueName: \"kubernetes.io/projected/3d6aba08-1940-4e35-8fc9-f1569e292f3a-kube-api-access-whjkb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:46:03.901462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.901435 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl"] Oct 30 15:46:03.903005 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:46:03.902977 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ddf09cc_81ce_4d72_9564_b1ed0704e183.slice/crio-603e62638dc5617fcceb15adcbc689b62f2e3873996e382395239020f52a1519 WatchSource:0}: Error finding container 603e62638dc5617fcceb15adcbc689b62f2e3873996e382395239020f52a1519: Status 404 returned error can't find the container with id 603e62638dc5617fcceb15adcbc689b62f2e3873996e382395239020f52a1519 Oct 30 15:46:03.943716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:03.943689 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm"] Oct 30 15:46:03.945512 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:46:03.945477 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0b1350_4d9c_4012_8761_ddccf2e06879.slice/crio-8615a9f0822d17df840cc54a4ae02473be4a99e08ef62f597c9eb3ad3af30b60 WatchSource:0}: Error finding container 8615a9f0822d17df840cc54a4ae02473be4a99e08ef62f597c9eb3ad3af30b60: Status 404 returned error can't find the container with id 8615a9f0822d17df840cc54a4ae02473be4a99e08ef62f597c9eb3ad3af30b60 Oct 30 15:46:04.571348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:04.571263 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" event={"ID":"1ddf09cc-81ce-4d72-9564-b1ed0704e183","Type":"ContainerStarted","Data":"603e62638dc5617fcceb15adcbc689b62f2e3873996e382395239020f52a1519"} Oct 30 15:46:04.574640 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:04.574604 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" event={"ID":"1b0b1350-4d9c-4012-8761-ddccf2e06879","Type":"ContainerStarted","Data":"1c76dba7632305cb0baf5c833ece875b9b092fa7500f9ece0ede9993b3ef5472"} Oct 30 15:46:04.574785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:04.574653 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" event={"ID":"1b0b1350-4d9c-4012-8761-ddccf2e06879","Type":"ContainerStarted","Data":"8615a9f0822d17df840cc54a4ae02473be4a99e08ef62f597c9eb3ad3af30b60"} Oct 30 15:46:04.575347 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:04.575324 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" Oct 30 15:46:04.674898 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:04.674815 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" podStartSLOduration=1.674797696 podStartE2EDuration="1.674797696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:46:04.671639144 +0000 UTC m=+473.370912302" watchObservedRunningTime="2026-10-30 15:46:04.674797696 +0000 UTC m=+473.374070853" Oct 30 15:46:05.428048 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:05.428012 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6aba08-1940-4e35-8fc9-f1569e292f3a" path="/var/lib/kubelet/pods/3d6aba08-1940-4e35-8fc9-f1569e292f3a/volumes" Oct 30 15:46:05.585858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:05.585757 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" containerName="manager" containerID="cri-o://da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6" gracePeriod=2 Oct 30 15:46:05.875591 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:05.875563 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:05.891606 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:05.891570 4401 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:06.033388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.033302 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f113622-3c9c-44ef-8079-7abaa0850d4e-kube-api-access-x8k46\" (UniqueName: \"kubernetes.io/projected/8f113622-3c9c-44ef-8079-7abaa0850d4e-kube-api-access-x8k46\") pod \"8f113622-3c9c-44ef-8079-7abaa0850d4e\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " Oct 30 15:46:06.033555 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.033493 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f113622-3c9c-44ef-8079-7abaa0850d4e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f113622-3c9c-44ef-8079-7abaa0850d4e-extensions-socket-volume\") pod \"8f113622-3c9c-44ef-8079-7abaa0850d4e\" (UID: \"8f113622-3c9c-44ef-8079-7abaa0850d4e\") " Oct 30 15:46:06.033784 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.033756 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f113622-3c9c-44ef-8079-7abaa0850d4e-extensions-socket-volume" pod "8f113622-3c9c-44ef-8079-7abaa0850d4e" (UID: "8f113622-3c9c-44ef-8079-7abaa0850d4e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:46:06.035929 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.035900 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f113622-3c9c-44ef-8079-7abaa0850d4e-kube-api-access-x8k46" pod "8f113622-3c9c-44ef-8079-7abaa0850d4e" (UID: "8f113622-3c9c-44ef-8079-7abaa0850d4e"). InnerVolumeSpecName "kube-api-access-x8k46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:46:06.135146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.135081 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8k46\" (UniqueName: \"kubernetes.io/projected/8f113622-3c9c-44ef-8079-7abaa0850d4e-kube-api-access-x8k46\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:46:06.135146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.135109 4401 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8f113622-3c9c-44ef-8079-7abaa0850d4e-extensions-socket-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:46:06.591871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.591775 4401 generic.go:356] "Generic (PLEG): container finished" podID="8f113622-3c9c-44ef-8079-7abaa0850d4e" containerID="da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6" exitCode=2 Oct 30 15:46:06.591871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.591835 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:06.591871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.591850 4401 scope.go:122] "RemoveContainer" containerID="da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6" Oct 30 15:46:06.593956 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.593931 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" event={"ID":"1ddf09cc-81ce-4d72-9564-b1ed0704e183","Type":"ContainerStarted","Data":"3b96ccf0a33af0b1da98c63e24d55a62e3c5f465fd7d808ab261eea56933058c"} Oct 30 15:46:06.601136 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.601119 4401 scope.go:122] "RemoveContainer" containerID="da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6" Oct 30 15:46:06.601378 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:06.601357 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6\": container with ID starting with da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6 not found: ID does not exist" containerID="da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6" Oct 30 15:46:06.601436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.601386 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6"} err="failed to get container status \"da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6\": rpc error: code = NotFound desc = could not find container \"da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6\": container with ID starting with da9efb4d4a42fc8a7ef8be34137efa0fb76aa298057416f6d6ca7b831a376ff6 not found: ID does not exist" Oct 30 15:46:06.607606 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:06.607581 4401 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:06.622984 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:06.622964 4401 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-kr5hb\" is forbidden: User \"system:node:ip-10-0-133-222.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-222.ec2.internal' and this object" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-kr5hb" Oct 30 15:46:06.705903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:06.705858 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" podStartSLOduration=1.453025635 podStartE2EDuration="3.705816578s" totalImagesPullingTime="2.252790943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:46:06.69093142 +0000 UTC m=+475.390204577" watchObservedRunningTime="2026-10-30 15:46:06.705816578 +0000 UTC m=+475.405089735" Oct 30 15:46:07.426280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:07.426249 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" path="/var/lib/kubelet/pods/8f113622-3c9c-44ef-8079-7abaa0850d4e/volumes" Oct 30 15:46:07.599657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:07.599629 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:16.596894 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:16.596862 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-dx5wm" Oct 30 15:46:18.606766 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:18.606732 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:21.965616 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:21.965584 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:46:31.575071 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.574994 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl"] Oct 30 15:46:31.575516 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.575197 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" podUID="1ddf09cc-81ce-4d72-9564-b1ed0704e183" containerName="manager" containerID="cri-o://3b96ccf0a33af0b1da98c63e24d55a62e3c5f465fd7d808ab261eea56933058c" gracePeriod=10 Oct 30 15:46:31.719238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.719206 4401 generic.go:356] "Generic (PLEG): container finished" podID="1ddf09cc-81ce-4d72-9564-b1ed0704e183" containerID="3b96ccf0a33af0b1da98c63e24d55a62e3c5f465fd7d808ab261eea56933058c" exitCode=0 Oct 30 15:46:31.719410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.719276 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" event={"ID":"1ddf09cc-81ce-4d72-9564-b1ed0704e183","Type":"ContainerDied","Data":"3b96ccf0a33af0b1da98c63e24d55a62e3c5f465fd7d808ab261eea56933058c"} Oct 30 15:46:31.815618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.815593 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:31.854986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.854963 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddf09cc-81ce-4d72-9564-b1ed0704e183-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddf09cc-81ce-4d72-9564-b1ed0704e183-extensions-socket-volume\") pod \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " Oct 30 15:46:31.855098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.855002 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddf09cc-81ce-4d72-9564-b1ed0704e183-kube-api-access-dmmm9\" (UniqueName: \"kubernetes.io/projected/1ddf09cc-81ce-4d72-9564-b1ed0704e183-kube-api-access-dmmm9\") pod \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\" (UID: \"1ddf09cc-81ce-4d72-9564-b1ed0704e183\") " Oct 30 15:46:31.855373 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.855351 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddf09cc-81ce-4d72-9564-b1ed0704e183-extensions-socket-volume" pod "1ddf09cc-81ce-4d72-9564-b1ed0704e183" (UID: "1ddf09cc-81ce-4d72-9564-b1ed0704e183"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:46:31.856989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.856968 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddf09cc-81ce-4d72-9564-b1ed0704e183-kube-api-access-dmmm9" pod "1ddf09cc-81ce-4d72-9564-b1ed0704e183" (UID: "1ddf09cc-81ce-4d72-9564-b1ed0704e183"). InnerVolumeSpecName "kube-api-access-dmmm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:46:31.956533 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.956499 4401 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1ddf09cc-81ce-4d72-9564-b1ed0704e183-extensions-socket-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:46:31.956533 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:31.956529 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmmm9\" (UniqueName: \"kubernetes.io/projected/1ddf09cc-81ce-4d72-9564-b1ed0704e183-kube-api-access-dmmm9\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:46:32.725945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:32.725912 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" event={"ID":"1ddf09cc-81ce-4d72-9564-b1ed0704e183","Type":"ContainerDied","Data":"603e62638dc5617fcceb15adcbc689b62f2e3873996e382395239020f52a1519"} Oct 30 15:46:32.726370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:32.725969 4401 scope.go:122] "RemoveContainer" containerID="3b96ccf0a33af0b1da98c63e24d55a62e3c5f465fd7d808ab261eea56933058c" Oct 30 15:46:32.726370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:32.725925 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl" Oct 30 15:46:32.804098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:32.804063 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl"] Oct 30 15:46:32.818210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:32.818174 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-8tqpl"] Oct 30 15:46:33.426136 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:33.426106 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddf09cc-81ce-4d72-9564-b1ed0704e183" path="/var/lib/kubelet/pods/1ddf09cc-81ce-4d72-9564-b1ed0704e183/volumes" Oct 30 15:46:35.964550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.964514 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f"] Oct 30 15:46:35.965141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.965090 4401 cpu_manager.go:414] "containerMap: removing container" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" containerName="manager" Oct 30 15:46:35.965141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.965109 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" containerName="manager" Oct 30 15:46:35.965141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.965130 4401 cpu_manager.go:414] "containerMap: removing container" podUID="1ddf09cc-81ce-4d72-9564-b1ed0704e183" containerName="manager" Oct 30 15:46:35.965141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.965139 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddf09cc-81ce-4d72-9564-b1ed0704e183" containerName="manager" Oct 30 15:46:35.965350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.965222 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f113622-3c9c-44ef-8079-7abaa0850d4e" containerName="manager" Oct 30 15:46:35.965350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.965235 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddf09cc-81ce-4d72-9564-b1ed0704e183" containerName="manager" Oct 30 15:46:35.968044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.968022 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:35.991524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:35.991502 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f"] Oct 30 15:46:36.037104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.037083 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x4s56\"" Oct 30 15:46:36.093028 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093001 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093037 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093057 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f64tt\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-kube-api-access-f64tt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093074 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093187 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093210 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093228 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093301 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.093457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.093333 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194019 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.193984 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194028 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f64tt\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-kube-api-access-f64tt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194252 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194210 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194331 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194260 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194331 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194302 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194335 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194440 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194555 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194470 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194689 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194734 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.194884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.194856 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.195076 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.195055 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.195167 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.195143 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.195295 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.195264 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.197053 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.197031 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.197170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.197153 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.219219 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.219163 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f64tt\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-kube-api-access-f64tt\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.220075 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.220059 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-njd6f\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.280593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.280568 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:36.599242 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.599215 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f"] Oct 30 15:46:36.599985 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:46:36.599957 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb364f9b9_6f62_4bd2_b1f8_9636e024529a.slice/crio-9cec31c6f67754ece8629d692dcb115903b439bc94590a7b3c67e9e2059d6aa8 WatchSource:0}: Error finding container 9cec31c6f67754ece8629d692dcb115903b439bc94590a7b3c67e9e2059d6aa8: Status 404 returned error can't find the container with id 9cec31c6f67754ece8629d692dcb115903b439bc94590a7b3c67e9e2059d6aa8 Oct 30 15:46:36.601936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.601901 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:46:36.602015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.601961 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:46:36.602015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.601989 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:46:36.748152 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.748113 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" event={"ID":"b364f9b9-6f62-4bd2-b1f8-9636e024529a","Type":"ContainerStarted","Data":"d741463923f2d50316e10d53e1f9192b6770b381cc2336d84508014b82070d89"} Oct 30 15:46:36.748340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.748184 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" event={"ID":"b364f9b9-6f62-4bd2-b1f8-9636e024529a","Type":"ContainerStarted","Data":"9cec31c6f67754ece8629d692dcb115903b439bc94590a7b3c67e9e2059d6aa8"} Oct 30 15:46:36.798951 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:36.798890 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podStartSLOduration=1.798872212 podStartE2EDuration="1.798872212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:46:36.796765161 +0000 UTC m=+505.496038320" watchObservedRunningTime="2026-10-30 15:46:36.798872212 +0000 UTC m=+505.498145370" Oct 30 15:46:37.281302 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:37.281272 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:37.285774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:37.285750 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:37.753163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:37.753130 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:37.754481 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:37.754460 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:46:51.965630 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:46:51.965589 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:46:52.150818 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.150780 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:46:52.153508 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.153488 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.173925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.173894 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:46:52.186162 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.186135 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 15:46:52.186336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.186209 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b9tr5\"" Oct 30 15:46:52.235527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.235453 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f6e42ec-f48a-4003-abb8-ce3d06af947a-config-file\") pod \"limitador-limitador-7c9b87d574-2qndw\" (UID: \"7f6e42ec-f48a-4003-abb8-ce3d06af947a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.235527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.235482 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbb5w\" (UniqueName: \"kubernetes.io/projected/7f6e42ec-f48a-4003-abb8-ce3d06af947a-kube-api-access-vbb5w\") pod \"limitador-limitador-7c9b87d574-2qndw\" (UID: \"7f6e42ec-f48a-4003-abb8-ce3d06af947a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.237758 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.237736 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:46:52.336596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.336562 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f6e42ec-f48a-4003-abb8-ce3d06af947a-config-file\") pod \"limitador-limitador-7c9b87d574-2qndw\" (UID: \"7f6e42ec-f48a-4003-abb8-ce3d06af947a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.336596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.336597 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbb5w\" (UniqueName: \"kubernetes.io/projected/7f6e42ec-f48a-4003-abb8-ce3d06af947a-kube-api-access-vbb5w\") pod \"limitador-limitador-7c9b87d574-2qndw\" (UID: \"7f6e42ec-f48a-4003-abb8-ce3d06af947a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.337214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.337194 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7f6e42ec-f48a-4003-abb8-ce3d06af947a-config-file\") pod \"limitador-limitador-7c9b87d574-2qndw\" (UID: \"7f6e42ec-f48a-4003-abb8-ce3d06af947a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.362582 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.362553 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbb5w\" (UniqueName: \"kubernetes.io/projected/7f6e42ec-f48a-4003-abb8-ce3d06af947a-kube-api-access-vbb5w\") pod \"limitador-limitador-7c9b87d574-2qndw\" (UID: \"7f6e42ec-f48a-4003-abb8-ce3d06af947a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.466144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.466110 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:52.769636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.769609 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:46:52.770545 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:46:52.770517 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6e42ec_f48a_4003_abb8_ce3d06af947a.slice/crio-5a634f20a11b456507b240013969b6cd5c3da5b3a2b9eb21ad73a98e09344482 WatchSource:0}: Error finding container 5a634f20a11b456507b240013969b6cd5c3da5b3a2b9eb21ad73a98e09344482: Status 404 returned error can't find the container with id 5a634f20a11b456507b240013969b6cd5c3da5b3a2b9eb21ad73a98e09344482 Oct 30 15:46:52.828070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:52.828040 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" event={"ID":"7f6e42ec-f48a-4003-abb8-ce3d06af947a","Type":"ContainerStarted","Data":"5a634f20a11b456507b240013969b6cd5c3da5b3a2b9eb21ad73a98e09344482"} Oct 30 15:46:53.834265 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:53.834164 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" event={"ID":"7f6e42ec-f48a-4003-abb8-ce3d06af947a","Type":"ContainerStarted","Data":"fa5a806ddd39a5ca775c0c6e1dd18548462bc317dbf8069d2a7b8c405608a75d"} Oct 30 15:46:53.834265 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:53.834237 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:46:53.880970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:46:53.880922 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" podStartSLOduration=1.095274268 podStartE2EDuration="1.880908079s" totalImagesPullingTime="785.633811ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:46:53.878990396 +0000 UTC m=+522.578263553" watchObservedRunningTime="2026-10-30 15:46:53.880908079 +0000 UTC m=+522.580181236" Oct 30 15:47:04.840173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:47:04.840099 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-2qndw" Oct 30 15:47:21.965205 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:47:21.965172 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:47:51.965297 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:47:51.965262 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:48:16.736157 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:16.736120 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:48:21.990372 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:48:21.990335 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:48:23.795979 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:23.795942 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-njd6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:48:23.796508 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:23.796004 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:48:26.310330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:26.310288 4401 generic.go:356] "Generic (PLEG): container finished" podID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerID="d741463923f2d50316e10d53e1f9192b6770b381cc2336d84508014b82070d89" exitCode=0 Oct 30 15:48:26.310765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:26.310359 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" event={"ID":"b364f9b9-6f62-4bd2-b1f8-9636e024529a","Type":"ContainerDied","Data":"d741463923f2d50316e10d53e1f9192b6770b381cc2336d84508014b82070d89"} Oct 30 15:48:26.310765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:26.310720 4401 scope.go:122] "RemoveContainer" containerID="d741463923f2d50316e10d53e1f9192b6770b381cc2336d84508014b82070d89" Oct 30 15:48:26.311856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:26.311809 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:48:26.311934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:26.311897 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:48:26.311934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:26.311929 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:48:27.317637 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:27.317604 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" event={"ID":"b364f9b9-6f62-4bd2-b1f8-9636e024529a","Type":"ContainerStarted","Data":"0720ca1dc0f92df33ca0e213ccc4cf14494723dc0c64f22450c49fd583fe8ad4"} Oct 30 15:48:28.281411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:28.281369 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:48:28.281606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:28.281593 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-njd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:48:28.281669 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:28.281648 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 15:48:29.282002 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:29.281961 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-njd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:48:29.282413 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:29.282030 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 15:48:30.282027 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:30.281987 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-njd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:48:30.282455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:30.282040 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 15:48:31.281294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:31.281257 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-njd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:48:31.281481 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:31.281310 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 15:48:32.281360 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:32.281319 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-njd6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:48:32.281750 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:32.281380 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 15:48:33.284980 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:33.284950 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:48:33.285394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:33.285242 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:48:33.285864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:48:33.285847 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:48:51.966195 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:48:51.966156 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:49:18.814915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:18.814874 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:49:21.077140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.077104 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6"] Oct 30 15:49:21.080746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.080724 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.106691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.106657 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 15:49:21.121140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.121107 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 15:49:21.121333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.121249 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 15:49:21.121333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.121250 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kvt8s\"" Oct 30 15:49:21.123449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.123423 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6"] Oct 30 15:49:21.154241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.154198 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.154437 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.154338 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.154437 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.154372 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjt5n\" (UniqueName: \"kubernetes.io/projected/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-kube-api-access-gjt5n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.154525 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.154494 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.154561 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.154535 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.154561 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.154550 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.255522 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.255487 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.255522 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.255525 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjt5n\" (UniqueName: \"kubernetes.io/projected/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-kube-api-access-gjt5n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.255774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.255563 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.255774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.255581 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.255774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.255594 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.255774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.255630 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.256023 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.256003 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.256023 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.256012 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.256124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.256074 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.257988 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.257965 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.258140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.258118 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.284856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.282915 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjt5n\" (UniqueName: \"kubernetes.io/projected/b4f4f6ca-7df0-428a-8b91-9e9301a154b4-kube-api-access-gjt5n\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8spw6\" (UID: \"b4f4f6ca-7df0-428a-8b91-9e9301a154b4\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.393854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.393796 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:21.711068 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.711042 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6"] Oct 30 15:49:21.713604 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:49:21.713572 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f4f6ca_7df0_428a_8b91_9e9301a154b4.slice/crio-07efee8f0a8868f8ce2389e7985c875383426521414212fb9c11502d570d4595 WatchSource:0}: Error finding container 07efee8f0a8868f8ce2389e7985c875383426521414212fb9c11502d570d4595: Status 404 returned error can't find the container with id 07efee8f0a8868f8ce2389e7985c875383426521414212fb9c11502d570d4595 Oct 30 15:49:21.715601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:21.715577 4401 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:49:21.979887 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:49:21.979773 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:49:22.585388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:22.585349 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" event={"ID":"b4f4f6ca-7df0-428a-8b91-9e9301a154b4","Type":"ContainerStarted","Data":"07efee8f0a8868f8ce2389e7985c875383426521414212fb9c11502d570d4595"} Oct 30 15:49:23.592790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:23.592746 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" event={"ID":"b4f4f6ca-7df0-428a-8b91-9e9301a154b4","Type":"ContainerStarted","Data":"5e1f04edf85205957c5c5839bac764b179094f75bf8c89f9d091b00b6f48c882"} Oct 30 15:49:23.593287 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:23.593266 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:49:23.642227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:23.642171 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" podStartSLOduration=1.537909105 podStartE2EDuration="2.642155268s" totalImagesPullingTime="1.104246163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:49:23.641186466 +0000 UTC m=+672.340459623" watchObservedRunningTime="2026-10-30 15:49:23.642155268 +0000 UTC m=+672.341428425" Oct 30 15:49:24.394999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.394903 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:24.605003 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.604968 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z"] Oct 30 15:49:24.609950 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.609924 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.631550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.631524 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 15:49:24.634766 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.634745 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z"] Oct 30 15:49:24.687670 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.687547 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.687670 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.687601 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.687670 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.687639 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.688028 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.687700 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.688028 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.687780 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzh42\" (UniqueName: \"kubernetes.io/projected/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-kube-api-access-bzh42\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.688028 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.687901 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.789063 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.788981 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.789379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.789040 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.789619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.789567 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.789850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.789791 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.790090 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.789975 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzh42\" (UniqueName: \"kubernetes.io/projected/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-kube-api-access-bzh42\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.790090 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.790042 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.791066 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.790954 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.791537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.791421 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.791922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.791812 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.809483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.809436 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.809601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.809502 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.820674 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.820647 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzh42\" (UniqueName: \"kubernetes.io/projected/3b8a62e2-b40a-4467-a128-46d2ba1d2fcc-kube-api-access-bzh42\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z\" (UID: \"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:24.925236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:24.925192 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:25.231606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:25.231576 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z"] Oct 30 15:49:25.232473 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:49:25.232448 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8a62e2_b40a_4467_a128_46d2ba1d2fcc.slice/crio-de8c4e6be8317a2d104c118c1cb7f28d3d19614744e01065a4e35ddacd26c3e6 WatchSource:0}: Error finding container de8c4e6be8317a2d104c118c1cb7f28d3d19614744e01065a4e35ddacd26c3e6: Status 404 returned error can't find the container with id de8c4e6be8317a2d104c118c1cb7f28d3d19614744e01065a4e35ddacd26c3e6 Oct 30 15:49:25.608813 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:25.608786 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" event={"ID":"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc","Type":"ContainerStarted","Data":"e33dc725d54d4b9b1343c816d9034ff9dcbceb4be5dfb114cdfe7ca9348e47ac"} Oct 30 15:49:25.609185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:25.608847 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" event={"ID":"3b8a62e2-b40a-4467-a128-46d2ba1d2fcc","Type":"ContainerStarted","Data":"de8c4e6be8317a2d104c118c1cb7f28d3d19614744e01065a4e35ddacd26c3e6"} Oct 30 15:49:25.659046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:25.659001 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" podStartSLOduration=1.439137503 podStartE2EDuration="1.658989399s" totalImagesPullingTime="219.851896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:49:25.656933577 +0000 UTC m=+674.356206734" watchObservedRunningTime="2026-10-30 15:49:25.658989399 +0000 UTC m=+674.358262555" Oct 30 15:49:25.926313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:25.926281 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:30.072583 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:30.072550 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:49:31.394590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:31.394537 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:31.408165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:31.408121 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:31.652812 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:31.652713 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8spw6" Oct 30 15:49:34.926405 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:34.926361 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:34.939305 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:34.939269 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:35.673938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:35.673909 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z" Oct 30 15:49:44.994730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:44.994691 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-z5ww4"] Oct 30 15:49:44.999085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:44.999064 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.015880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.015853 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-z5ww4"] Oct 30 15:49:45.037106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.037072 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 15:49:45.037456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.037434 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 15:49:45.037768 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.037749 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-9d6hx\"" Oct 30 15:49:45.178614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.178583 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f3ce03-e0e4-4703-8943-58db11f3967e-metrics-certs\") pod \"ai-gateway-controller-7489558957-z5ww4\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.178819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.178744 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csfxp\" (UniqueName: \"kubernetes.io/projected/72f3ce03-e0e4-4703-8943-58db11f3967e-kube-api-access-csfxp\") pod \"ai-gateway-controller-7489558957-z5ww4\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.279977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.279899 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f3ce03-e0e4-4703-8943-58db11f3967e-metrics-certs\") pod \"ai-gateway-controller-7489558957-z5ww4\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.280120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.280002 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csfxp\" (UniqueName: \"kubernetes.io/projected/72f3ce03-e0e4-4703-8943-58db11f3967e-kube-api-access-csfxp\") pod \"ai-gateway-controller-7489558957-z5ww4\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.282272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.282244 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f3ce03-e0e4-4703-8943-58db11f3967e-metrics-certs\") pod \"ai-gateway-controller-7489558957-z5ww4\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.305200 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.305170 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csfxp\" (UniqueName: \"kubernetes.io/projected/72f3ce03-e0e4-4703-8943-58db11f3967e-kube-api-access-csfxp\") pod \"ai-gateway-controller-7489558957-z5ww4\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.313845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.313812 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:45.636491 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.636461 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-z5ww4"] Oct 30 15:49:45.638037 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:49:45.638013 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f3ce03_e0e4_4703_8943_58db11f3967e.slice/crio-2e92eaf1519951021102dfd8f5384589657f4f67ae01c21ee7946f18720eb22d WatchSource:0}: Error finding container 2e92eaf1519951021102dfd8f5384589657f4f67ae01c21ee7946f18720eb22d: Status 404 returned error can't find the container with id 2e92eaf1519951021102dfd8f5384589657f4f67ae01c21ee7946f18720eb22d Oct 30 15:49:45.714650 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:45.714618 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" event={"ID":"72f3ce03-e0e4-4703-8943-58db11f3967e","Type":"ContainerStarted","Data":"2e92eaf1519951021102dfd8f5384589657f4f67ae01c21ee7946f18720eb22d"} Oct 30 15:49:48.736902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:48.736863 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" event={"ID":"72f3ce03-e0e4-4703-8943-58db11f3967e","Type":"ContainerStarted","Data":"ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef"} Oct 30 15:49:48.737304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:48.737051 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:49:48.782454 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:48.782397 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" podStartSLOduration=2.5575639839999997 podStartE2EDuration="4.782385023s" totalImagesPullingTime="2.224821039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:49:48.78090417 +0000 UTC m=+697.480177328" watchObservedRunningTime="2026-10-30 15:49:48.782385023 +0000 UTC m=+697.481658180" Oct 30 15:49:50.249161 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.249128 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h"] Oct 30 15:49:50.252880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.252852 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.270749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.270719 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h"] Oct 30 15:49:50.282267 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.282245 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 15:49:50.282445 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.282430 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tpd2q\"" Oct 30 15:49:50.325222 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.325184 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0567028d-7000-45f9-ab89-c4524254517c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6tt6h\" (UID: \"0567028d-7000-45f9-ab89-c4524254517c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.325408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.325375 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnp97\" (UniqueName: \"kubernetes.io/projected/0567028d-7000-45f9-ab89-c4524254517c-kube-api-access-dnp97\") pod \"payload-pre-processing-7b98bb678c-6tt6h\" (UID: \"0567028d-7000-45f9-ab89-c4524254517c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.392559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.392511 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bfqbb"] Oct 30 15:49:50.396147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.396114 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.426125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.426101 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0567028d-7000-45f9-ab89-c4524254517c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6tt6h\" (UID: \"0567028d-7000-45f9-ab89-c4524254517c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.426272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.426131 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnp97\" (UniqueName: \"kubernetes.io/projected/0567028d-7000-45f9-ab89-c4524254517c-kube-api-access-dnp97\") pod \"payload-pre-processing-7b98bb678c-6tt6h\" (UID: \"0567028d-7000-45f9-ab89-c4524254517c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.426272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.426152 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlbpw\" (UniqueName: \"kubernetes.io/projected/b1a28f78-9ced-4190-9c1f-413f206fd8c5-kube-api-access-vlbpw\") pod \"payload-processing-5f78687d5b-bfqbb\" (UID: \"b1a28f78-9ced-4190-9c1f-413f206fd8c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.426378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.426281 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a28f78-9ced-4190-9c1f-413f206fd8c5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bfqbb\" (UID: \"b1a28f78-9ced-4190-9c1f-413f206fd8c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.426872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.426850 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0567028d-7000-45f9-ab89-c4524254517c-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6tt6h\" (UID: \"0567028d-7000-45f9-ab89-c4524254517c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.432965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.432943 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bfqbb"] Oct 30 15:49:50.471398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.471373 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnp97\" (UniqueName: \"kubernetes.io/projected/0567028d-7000-45f9-ab89-c4524254517c-kube-api-access-dnp97\") pod \"payload-pre-processing-7b98bb678c-6tt6h\" (UID: \"0567028d-7000-45f9-ab89-c4524254517c\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.527179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.527104 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlbpw\" (UniqueName: \"kubernetes.io/projected/b1a28f78-9ced-4190-9c1f-413f206fd8c5-kube-api-access-vlbpw\") pod \"payload-processing-5f78687d5b-bfqbb\" (UID: \"b1a28f78-9ced-4190-9c1f-413f206fd8c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.527311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.527191 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a28f78-9ced-4190-9c1f-413f206fd8c5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bfqbb\" (UID: \"b1a28f78-9ced-4190-9c1f-413f206fd8c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.527860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.527819 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1a28f78-9ced-4190-9c1f-413f206fd8c5-plugins-config-volume\") pod \"payload-processing-5f78687d5b-bfqbb\" (UID: \"b1a28f78-9ced-4190-9c1f-413f206fd8c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.552603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.552573 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlbpw\" (UniqueName: \"kubernetes.io/projected/b1a28f78-9ced-4190-9c1f-413f206fd8c5-kube-api-access-vlbpw\") pod \"payload-processing-5f78687d5b-bfqbb\" (UID: \"b1a28f78-9ced-4190-9c1f-413f206fd8c5\") " pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.565464 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.565441 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:50.710729 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.710696 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:50.874390 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:50.874366 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h"] Oct 30 15:49:50.875563 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:49:50.875537 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0567028d_7000_45f9_ab89_c4524254517c.slice/crio-94ea7e9e9bbf1e1422ee8b98c7fba65f986bde04386e5819d05d5e2e3d317aa0 WatchSource:0}: Error finding container 94ea7e9e9bbf1e1422ee8b98c7fba65f986bde04386e5819d05d5e2e3d317aa0: Status 404 returned error can't find the container with id 94ea7e9e9bbf1e1422ee8b98c7fba65f986bde04386e5819d05d5e2e3d317aa0 Oct 30 15:49:51.011323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:51.011296 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-bfqbb"] Oct 30 15:49:51.012708 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:49:51.012683 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a28f78_9ced_4190_9c1f_413f206fd8c5.slice/crio-5d7a6fa05346a3cc66fb968d5e3aa53cee5c1f0446adea5d22d7bbc0cda7c3db WatchSource:0}: Error finding container 5d7a6fa05346a3cc66fb968d5e3aa53cee5c1f0446adea5d22d7bbc0cda7c3db: Status 404 returned error can't find the container with id 5d7a6fa05346a3cc66fb968d5e3aa53cee5c1f0446adea5d22d7bbc0cda7c3db Oct 30 15:49:51.756052 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:51.756006 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" event={"ID":"b1a28f78-9ced-4190-9c1f-413f206fd8c5","Type":"ContainerStarted","Data":"5d7a6fa05346a3cc66fb968d5e3aa53cee5c1f0446adea5d22d7bbc0cda7c3db"} Oct 30 15:49:51.757573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:51.757537 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" event={"ID":"0567028d-7000-45f9-ab89-c4524254517c","Type":"ContainerStarted","Data":"94ea7e9e9bbf1e1422ee8b98c7fba65f986bde04386e5819d05d5e2e3d317aa0"} Oct 30 15:49:51.979201 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:49:51.979168 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:49:52.005006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.004604 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj"] Oct 30 15:49:52.010644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.010570 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.030692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.030664 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj"] Oct 30 15:49:52.043238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043203 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043254 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043374 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a4faf0-b580-4e3b-b71c-cda031c990f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043517 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043574 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043608 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043676 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043710 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.043862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.043853 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mmrf\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-kube-api-access-8mmrf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145220 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145185 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mmrf\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-kube-api-access-8mmrf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145255 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145282 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145319 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a4faf0-b580-4e3b-b71c-cda031c990f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145358 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145387 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145406 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145671 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145429 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.145671 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.145452 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.146186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.146150 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.146323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.146222 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.146323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.146304 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.146468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.146407 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.146520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.146462 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a4faf0-b580-4e3b-b71c-cda031c990f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.148507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.148487 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.148773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.148754 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.171906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.171876 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.172293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.172273 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mmrf\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-kube-api-access-8mmrf\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-fqjmj\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.337042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.336950 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:52.675296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.675268 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj"] Oct 30 15:49:52.678810 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:49:52.678777 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a4faf0_b580_4e3b_b71c_cda031c990f9.slice/crio-c5076aa50392bcf2f37f0d2eb57fff86c1b39e9ccc2bac13061bc3d8182ceac0 WatchSource:0}: Error finding container c5076aa50392bcf2f37f0d2eb57fff86c1b39e9ccc2bac13061bc3d8182ceac0: Status 404 returned error can't find the container with id c5076aa50392bcf2f37f0d2eb57fff86c1b39e9ccc2bac13061bc3d8182ceac0 Oct 30 15:49:52.681725 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.681686 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:49:52.681856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.681766 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:49:52.681856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.681799 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:49:52.766072 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:52.766041 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" event={"ID":"80a4faf0-b580-4e3b-b71c-cda031c990f9","Type":"ContainerStarted","Data":"c5076aa50392bcf2f37f0d2eb57fff86c1b39e9ccc2bac13061bc3d8182ceac0"} Oct 30 15:49:53.777038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.777001 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" event={"ID":"80a4faf0-b580-4e3b-b71c-cda031c990f9","Type":"ContainerStarted","Data":"a6b23b5dd1ee838c037e2c8766a87800e755a8149d1a2c10eda9fb4574b2242d"} Oct 30 15:49:53.778628 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.778600 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" event={"ID":"b1a28f78-9ced-4190-9c1f-413f206fd8c5","Type":"ContainerStarted","Data":"553236384f120ca9d5f6017ce25c10a7704da3c392272dd744ddd87885e5e27c"} Oct 30 15:49:53.778785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.778699 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:49:53.780049 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.780026 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" event={"ID":"0567028d-7000-45f9-ab89-c4524254517c","Type":"ContainerStarted","Data":"84123cfeb59f0773239c897bb8be7c97fede411e42659fec9697140e2487d2d3"} Oct 30 15:49:53.780163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.780068 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:49:53.841661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.841619 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podStartSLOduration=2.841608311 podStartE2EDuration="2.841608311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:49:53.839876049 +0000 UTC m=+702.539149205" watchObservedRunningTime="2026-10-30 15:49:53.841608311 +0000 UTC m=+702.540881467" Oct 30 15:49:53.895156 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.895105 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" podStartSLOduration=1.4788979740000001 podStartE2EDuration="3.89509322s" totalImagesPullingTime="2.416195246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:49:53.893971286 +0000 UTC m=+702.593244443" watchObservedRunningTime="2026-10-30 15:49:53.89509322 +0000 UTC m=+702.594366371" Oct 30 15:49:53.939880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:53.939803 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" podStartSLOduration=1.5104467929999998 podStartE2EDuration="3.939790935s" totalImagesPullingTime="2.429344142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:49:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:49:53.939016109 +0000 UTC m=+702.638289266" watchObservedRunningTime="2026-10-30 15:49:53.939790935 +0000 UTC m=+702.639064092" Oct 30 15:49:54.337558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:54.337524 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:49:54.339063 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:54.339035 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 15:49:54.339199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:54.339109 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 15:49:55.338373 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:55.338330 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 15:49:55.338805 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:55.338403 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 15:49:56.338091 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:56.338053 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 15:49:56.338274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:56.338109 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 15:49:57.337537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:57.337503 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 15:49:57.337958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:57.337554 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 15:49:58.337538 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:58.337502 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 15:49:58.337956 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:58.337554 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 15:49:59.338058 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:59.338017 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 15:49:59.338473 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:59.338088 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 15:49:59.746000 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:49:59.745964 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:50:00.042577 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.042500 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-54d48c9567-8qzkb"] Oct 30 15:50:00.048553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.048532 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.066573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.066534 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54d48c9567-8qzkb"] Oct 30 15:50:00.122598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.122558 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kbgt\" (UniqueName: \"kubernetes.io/projected/8803c379-9a94-4756-adae-baa7172b3212-kube-api-access-2kbgt\") pod \"ai-gateway-controller-54d48c9567-8qzkb\" (UID: \"8803c379-9a94-4756-adae-baa7172b3212\") " pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.122598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.122597 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8803c379-9a94-4756-adae-baa7172b3212-metrics-certs\") pod \"ai-gateway-controller-54d48c9567-8qzkb\" (UID: \"8803c379-9a94-4756-adae-baa7172b3212\") " pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.223472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.223433 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kbgt\" (UniqueName: \"kubernetes.io/projected/8803c379-9a94-4756-adae-baa7172b3212-kube-api-access-2kbgt\") pod \"ai-gateway-controller-54d48c9567-8qzkb\" (UID: \"8803c379-9a94-4756-adae-baa7172b3212\") " pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.223472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.223475 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8803c379-9a94-4756-adae-baa7172b3212-metrics-certs\") pod \"ai-gateway-controller-54d48c9567-8qzkb\" (UID: \"8803c379-9a94-4756-adae-baa7172b3212\") " pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.225851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.225801 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8803c379-9a94-4756-adae-baa7172b3212-metrics-certs\") pod \"ai-gateway-controller-54d48c9567-8qzkb\" (UID: \"8803c379-9a94-4756-adae-baa7172b3212\") " pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.249414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.249379 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kbgt\" (UniqueName: \"kubernetes.io/projected/8803c379-9a94-4756-adae-baa7172b3212-kube-api-access-2kbgt\") pod \"ai-gateway-controller-54d48c9567-8qzkb\" (UID: \"8803c379-9a94-4756-adae-baa7172b3212\") " pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.362130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.362098 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:00.667554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.667500 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-54d48c9567-8qzkb"] Oct 30 15:50:00.669278 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:00.669251 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8803c379_9a94_4756_adae_baa7172b3212.slice/crio-ece4ea5e14c4facd47232644e17f946b2c5baaac3995be5179c87539ccaedfe6 WatchSource:0}: Error finding container ece4ea5e14c4facd47232644e17f946b2c5baaac3995be5179c87539ccaedfe6: Status 404 returned error can't find the container with id ece4ea5e14c4facd47232644e17f946b2c5baaac3995be5179c87539ccaedfe6 Oct 30 15:50:00.818632 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:00.818588 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" event={"ID":"8803c379-9a94-4756-adae-baa7172b3212","Type":"ContainerStarted","Data":"ece4ea5e14c4facd47232644e17f946b2c5baaac3995be5179c87539ccaedfe6"} Oct 30 15:50:01.043323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:01.043241 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:50:01.043620 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:01.043593 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:50:01.044457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:01.044439 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:50:01.825303 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:01.825263 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" event={"ID":"8803c379-9a94-4756-adae-baa7172b3212","Type":"ContainerStarted","Data":"c6ee3bf42d2ca694a9efb6879acf3ea2457d4e15085f76c51e38dd99374fe800"} Oct 30 15:50:01.825730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:01.825508 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:01.869541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:01.869482 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" podStartSLOduration=1.4762420139999999 podStartE2EDuration="1.86946791s" totalImagesPullingTime="393.225896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:01.868525577 +0000 UTC m=+710.567798735" watchObservedRunningTime="2026-10-30 15:50:01.86946791 +0000 UTC m=+710.568741067" Oct 30 15:50:04.789124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:04.789043 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6tt6h" Oct 30 15:50:04.789124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:04.789122 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-bfqbb" Oct 30 15:50:07.041456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.041424 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26"] Oct 30 15:50:07.045381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.045356 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081496 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081462 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c5b157-a5fd-4560-b25f-25719b24fef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081546 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081620 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081636 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081767 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081667 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081831 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081803 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081854 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.081942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.081880 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbq5\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-kube-api-access-hbbq5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.082029 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.082012 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.082843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.082809 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26"] Oct 30 15:50:07.183156 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183124 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183188 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183211 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183232 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183367 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183408 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183431 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbq5\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-kube-api-access-hbbq5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183506 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183708 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183571 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c5b157-a5fd-4560-b25f-25719b24fef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183708 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183596 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183708 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183626 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183744 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.183908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.183862 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.184166 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.184146 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c5b157-a5fd-4560-b25f-25719b24fef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.185642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.185619 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.185875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.185857 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.208895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.208866 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.208999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.208985 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbq5\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-kube-api-access-hbbq5\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-hcj26\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.359104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.359071 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:07.677483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.677456 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26"] Oct 30 15:50:07.678919 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:07.678888 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c5b157_a5fd_4560_b25f_25719b24fef7.slice/crio-2cc84536a1a2d61c78f12f7fad3cfe190a68c7e466769ccd0a36515d16003d02 WatchSource:0}: Error finding container 2cc84536a1a2d61c78f12f7fad3cfe190a68c7e466769ccd0a36515d16003d02: Status 404 returned error can't find the container with id 2cc84536a1a2d61c78f12f7fad3cfe190a68c7e466769ccd0a36515d16003d02 Oct 30 15:50:07.681096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.681064 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:07.681193 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.681176 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:07.681240 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.681215 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:07.856522 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.856486 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" event={"ID":"29c5b157-a5fd-4560-b25f-25719b24fef7","Type":"ContainerStarted","Data":"1308ddb569752d302b4db1c4b5bc92a11b3cf65fcca61a903fa92df357f4b59e"} Oct 30 15:50:07.856893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.856538 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" event={"ID":"29c5b157-a5fd-4560-b25f-25719b24fef7","Type":"ContainerStarted","Data":"2cc84536a1a2d61c78f12f7fad3cfe190a68c7e466769ccd0a36515d16003d02"} Oct 30 15:50:07.906584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:07.906535 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podStartSLOduration=0.906522214 podStartE2EDuration="906.522214ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:07.904654736 +0000 UTC m=+716.603927894" watchObservedRunningTime="2026-10-30 15:50:07.906522214 +0000 UTC m=+716.605795371" Oct 30 15:50:08.369847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:08.364903 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:08.369847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:08.368953 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hcj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 15:50:08.369847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:08.369018 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 15:50:09.363673 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:09.363221 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hcj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 15:50:09.363673 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:09.363284 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 15:50:10.369965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:10.369923 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hcj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 15:50:10.370550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:10.370516 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 15:50:11.364848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:11.360227 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hcj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 15:50:11.364848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:11.360304 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 15:50:12.360318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:12.360275 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hcj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 15:50:12.360977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:12.360941 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 15:50:12.832303 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:12.832225 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-54d48c9567-8qzkb" Oct 30 15:50:12.910523 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:12.910488 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-z5ww4"] Oct 30 15:50:12.910716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:12.910672 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" podUID="72f3ce03-e0e4-4703-8943-58db11f3967e" containerName="manager" containerID="cri-o://ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef" gracePeriod=30 Oct 30 15:50:13.204888 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.204864 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:50:13.276255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.276217 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f3ce03-e0e4-4703-8943-58db11f3967e-kube-api-access-csfxp\" (UniqueName: \"kubernetes.io/projected/72f3ce03-e0e4-4703-8943-58db11f3967e-kube-api-access-csfxp\") pod \"72f3ce03-e0e4-4703-8943-58db11f3967e\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " Oct 30 15:50:13.276470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.276444 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f3ce03-e0e4-4703-8943-58db11f3967e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f3ce03-e0e4-4703-8943-58db11f3967e-metrics-certs\") pod \"72f3ce03-e0e4-4703-8943-58db11f3967e\" (UID: \"72f3ce03-e0e4-4703-8943-58db11f3967e\") " Oct 30 15:50:13.278715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.278673 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f3ce03-e0e4-4703-8943-58db11f3967e-metrics-certs" pod "72f3ce03-e0e4-4703-8943-58db11f3967e" (UID: "72f3ce03-e0e4-4703-8943-58db11f3967e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:50:13.278857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.278668 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f3ce03-e0e4-4703-8943-58db11f3967e-kube-api-access-csfxp" pod "72f3ce03-e0e4-4703-8943-58db11f3967e" (UID: "72f3ce03-e0e4-4703-8943-58db11f3967e"). InnerVolumeSpecName "kube-api-access-csfxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:50:13.360675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.360189 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hcj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 15:50:13.360675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.360254 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 15:50:13.377709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.377664 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csfxp\" (UniqueName: \"kubernetes.io/projected/72f3ce03-e0e4-4703-8943-58db11f3967e-kube-api-access-csfxp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:50:13.377709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.377706 4401 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72f3ce03-e0e4-4703-8943-58db11f3967e-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:50:13.904690 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.904649 4401 generic.go:356] "Generic (PLEG): container finished" podID="72f3ce03-e0e4-4703-8943-58db11f3967e" containerID="ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef" exitCode=0 Oct 30 15:50:13.904876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.904714 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" Oct 30 15:50:13.904876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.904729 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" event={"ID":"72f3ce03-e0e4-4703-8943-58db11f3967e","Type":"ContainerDied","Data":"ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef"} Oct 30 15:50:13.904876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.904803 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7489558957-z5ww4" event={"ID":"72f3ce03-e0e4-4703-8943-58db11f3967e","Type":"ContainerDied","Data":"2e92eaf1519951021102dfd8f5384589657f4f67ae01c21ee7946f18720eb22d"} Oct 30 15:50:13.904876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.904838 4401 scope.go:122] "RemoveContainer" containerID="ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef" Oct 30 15:50:13.913978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.913690 4401 scope.go:122] "RemoveContainer" containerID="ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef" Oct 30 15:50:13.914045 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:50:13.914018 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef\": container with ID starting with ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef not found: ID does not exist" containerID="ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef" Oct 30 15:50:13.914085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.914041 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef"} err="failed to get container status \"ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef\": rpc error: code = NotFound desc = could not find container \"ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef\": container with ID starting with ecb55bac8b6a37c91d108ea9158ce6ad2086ea266daa7244fc582fb15e5756ef not found: ID does not exist" Oct 30 15:50:13.963752 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.963725 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-z5ww4"] Oct 30 15:50:13.968126 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:13.968090 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7489558957-z5ww4"] Oct 30 15:50:14.360388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:14.360360 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-hcj26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 15:50:14.360557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:14.360416 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 15:50:15.428058 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:15.428016 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f3ce03-e0e4-4703-8943-58db11f3967e" path="/var/lib/kubelet/pods/72f3ce03-e0e4-4703-8943-58db11f3967e/volumes" Oct 30 15:50:15.936779 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:15.936745 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:15.937204 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:15.937149 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:15.937985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:15.937963 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:50:21.966112 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:50:21.966069 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:50:40.439461 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.439428 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2"] Oct 30 15:50:40.439980 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.439888 4401 cpu_manager.go:414] "containerMap: removing container" podUID="72f3ce03-e0e4-4703-8943-58db11f3967e" containerName="manager" Oct 30 15:50:40.439980 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.439914 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f3ce03-e0e4-4703-8943-58db11f3967e" containerName="manager" Oct 30 15:50:40.440066 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.440021 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f3ce03-e0e4-4703-8943-58db11f3967e" containerName="manager" Oct 30 15:50:40.443460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.443441 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.476349 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.476321 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2"] Oct 30 15:50:40.477122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.477097 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-040f9a-openshift-default-dockercfg-r5zts\"" Oct 30 15:50:40.541006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.540970 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541112 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541159 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d05b27-3f87-4a35-8502-3ddedaa5456e-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541278 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541326 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541386 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541423 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnwc\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-kube-api-access-bgnwc\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541539 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541522 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.541599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.541558 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.642734 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.642697 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.642960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.642746 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.642960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.642808 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.642960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.642887 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.642960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.642915 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d05b27-3f87-4a35-8502-3ddedaa5456e-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.642960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.642955 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.643247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.642983 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.643247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.643028 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.643247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.643052 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnwc\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-kube-api-access-bgnwc\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.643602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.643577 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.643709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.643621 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.643898 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.643877 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.644054 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.644031 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d05b27-3f87-4a35-8502-3ddedaa5456e-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.644202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.644182 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.645544 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.645523 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.645625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.645540 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.679094 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.679066 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.679748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.679723 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnwc\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-kube-api-access-bgnwc\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.755972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.755895 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:40.898727 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.898684 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj"] Oct 30 15:50:40.903670 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.903644 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.927938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.927913 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a2b3f1-openshift-default-dockercfg-jmzld\"" Oct 30 15:50:40.938315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.938285 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj"] Oct 30 15:50:40.945998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.945973 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946013 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946040 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946209 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946191 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946248 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946240 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb734159-1499-4633-9a9e-aefdfcf9b76f-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946289 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946277 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946345 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946434 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vn4h\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-kube-api-access-6vn4h\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:40.946533 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:40.946519 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047570 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047489 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047570 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047529 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047570 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047554 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047674 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047730 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047789 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb734159-1499-4633-9a9e-aefdfcf9b76f-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047863 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.047901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047895 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.048153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047946 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vn4h\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-kube-api-access-6vn4h\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.048153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.047984 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.048153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.048097 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.048393 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.048371 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.048456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.048432 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.048595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.048573 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb734159-1499-4633-9a9e-aefdfcf9b76f-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.049858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.049818 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.050217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.050197 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.063393 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.063367 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd"] Oct 30 15:50:41.068470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.068444 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.084776 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.084752 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.095135 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.095108 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vn4h\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-kube-api-access-6vn4h\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.099965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.097757 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2"] Oct 30 15:50:41.099965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.099099 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-9e2b29-openshift-default-dockercfg-vfgdd\"" Oct 30 15:50:41.099965 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:41.099731 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d05b27_3f87_4a35_8502_3ddedaa5456e.slice/crio-7b3b177b4e4b1b9389e79d5b8c1b7b7c86d0d13b088244ee114a828daf4d7137 WatchSource:0}: Error finding container 7b3b177b4e4b1b9389e79d5b8c1b7b7c86d0d13b088244ee114a828daf4d7137: Status 404 returned error can't find the container with id 7b3b177b4e4b1b9389e79d5b8c1b7b7c86d0d13b088244ee114a828daf4d7137 Oct 30 15:50:41.101431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.101408 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd"] Oct 30 15:50:41.102680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.102651 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.102775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.102719 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.102775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.102764 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.149509 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149484 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149527 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149601 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149701 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149652 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149701 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149698 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149727 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149894 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149802 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149891 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.149986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.149966 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqmbr\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-kube-api-access-cqmbr\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.218756 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.218723 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:41.251357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251327 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251529 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251375 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqmbr\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-kube-api-access-cqmbr\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251529 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251428 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251529 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251449 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251529 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251488 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251537 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251571 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251613 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.251757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251651 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.252005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.251803 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.252145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.252126 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.255900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.255144 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.255900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.255207 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.255900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.255773 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.256639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.256592 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.257290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.257264 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.292078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.289070 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.293377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.293350 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqmbr\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-kube-api-access-cqmbr\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.383524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.383488 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:41.555663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.555633 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj"] Oct 30 15:50:41.558904 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:41.558878 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb734159_1499_4633_9a9e_aefdfcf9b76f.slice/crio-4ac05b280756b1dfb097be8609382e10ad61a7ab9c750048c47288d30e50a192 WatchSource:0}: Error finding container 4ac05b280756b1dfb097be8609382e10ad61a7ab9c750048c47288d30e50a192: Status 404 returned error can't find the container with id 4ac05b280756b1dfb097be8609382e10ad61a7ab9c750048c47288d30e50a192 Oct 30 15:50:41.565035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.564990 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.565157 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.565078 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.565157 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.565111 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.729576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.729548 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd"] Oct 30 15:50:41.730220 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:41.730197 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod564bbe1a_6e89_4b9f_9432_795b6205f0a8.slice/crio-fbcb1a367744fa7fb8e02b3a387cdf1574e4a36bd61a4469368316567b71e879 WatchSource:0}: Error finding container fbcb1a367744fa7fb8e02b3a387cdf1574e4a36bd61a4469368316567b71e879: Status 404 returned error can't find the container with id fbcb1a367744fa7fb8e02b3a387cdf1574e4a36bd61a4469368316567b71e879 Oct 30 15:50:41.733342 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.733217 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.733342 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.733293 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:41.733342 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:41.733322 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:50:42.056359 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.056264 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" event={"ID":"fb734159-1499-4633-9a9e-aefdfcf9b76f","Type":"ContainerStarted","Data":"beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b"} Oct 30 15:50:42.056359 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.056328 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" event={"ID":"fb734159-1499-4633-9a9e-aefdfcf9b76f","Type":"ContainerStarted","Data":"4ac05b280756b1dfb097be8609382e10ad61a7ab9c750048c47288d30e50a192"} Oct 30 15:50:42.057958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.057936 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" event={"ID":"564bbe1a-6e89-4b9f-9432-795b6205f0a8","Type":"ContainerStarted","Data":"cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988"} Oct 30 15:50:42.058096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.057976 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" event={"ID":"564bbe1a-6e89-4b9f-9432-795b6205f0a8","Type":"ContainerStarted","Data":"fbcb1a367744fa7fb8e02b3a387cdf1574e4a36bd61a4469368316567b71e879"} Oct 30 15:50:42.059471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.059451 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" event={"ID":"25d05b27-3f87-4a35-8502-3ddedaa5456e","Type":"ContainerStarted","Data":"6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5"} Oct 30 15:50:42.059563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.059485 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" event={"ID":"25d05b27-3f87-4a35-8502-3ddedaa5456e","Type":"ContainerStarted","Data":"7b3b177b4e4b1b9389e79d5b8c1b7b7c86d0d13b088244ee114a828daf4d7137"} Oct 30 15:50:42.107511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.107466 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podStartSLOduration=2.107452602 podStartE2EDuration="2.107452602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:42.106063527 +0000 UTC m=+750.805336686" watchObservedRunningTime="2026-10-30 15:50:42.107452602 +0000 UTC m=+750.806725759" Oct 30 15:50:42.156029 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.155969 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" podStartSLOduration=2.155950468 podStartE2EDuration="2.155950468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:42.154187311 +0000 UTC m=+750.853460467" watchObservedRunningTime="2026-10-30 15:50:42.155950468 +0000 UTC m=+750.855223628" Oct 30 15:50:42.207247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.207203 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podStartSLOduration=1.207190587 podStartE2EDuration="1.207190587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:42.204017874 +0000 UTC m=+750.903291042" watchObservedRunningTime="2026-10-30 15:50:42.207190587 +0000 UTC m=+750.906463722" Oct 30 15:50:42.219258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.219231 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:42.224435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.224412 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:42.384030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.383993 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:42.389232 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.389208 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:42.756152 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.756073 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:42.761323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:42.761301 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:43.065699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:43.065592 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:43.065699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:43.065661 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:43.065699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:43.065687 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:43.066432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:43.066412 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:50:43.066615 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:43.066597 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:50:43.066698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:43.066661 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:50:46.505507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.504202 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h"] Oct 30 15:50:46.510912 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.510889 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.525579 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.525553 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h"] Oct 30 15:50:46.536978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.536954 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ba9db9-dockercfg-6rtms\"" Oct 30 15:50:46.536978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.536959 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ba9db9\"" Oct 30 15:50:46.573050 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.573022 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9"] Oct 30 15:50:46.576740 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.576724 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:46.598891 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.595260 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9"] Oct 30 15:50:46.617644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.617391 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfb6\" (UniqueName: \"kubernetes.io/projected/72b2a776-bd23-41f3-908d-81cfc68ffb47-kube-api-access-zkfb6\") pod \"payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.617644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.617446 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2a776-bd23-41f3-908d-81cfc68ffb47-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.719042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.719007 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:46.719229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.719057 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfb6\" (UniqueName: \"kubernetes.io/projected/72b2a776-bd23-41f3-908d-81cfc68ffb47-kube-api-access-zkfb6\") pod \"payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.719229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.719082 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2a776-bd23-41f3-908d-81cfc68ffb47-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.719329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.719224 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgjds\" (UniqueName: \"kubernetes.io/projected/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-kube-api-access-qgjds\") pod \"payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:46.719733 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.719716 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2a776-bd23-41f3-908d-81cfc68ffb47-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.744778 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.744746 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfb6\" (UniqueName: \"kubernetes.io/projected/72b2a776-bd23-41f3-908d-81cfc68ffb47-kube-api-access-zkfb6\") pod \"payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.820536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.820452 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgjds\" (UniqueName: \"kubernetes.io/projected/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-kube-api-access-qgjds\") pod \"payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:46.820536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.820517 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:46.821261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.821241 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:46.825241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.825214 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:46.845783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.845762 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgjds\" (UniqueName: \"kubernetes.io/projected/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-kube-api-access-qgjds\") pod \"payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:46.893421 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:46.893389 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:47.128819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:47.128794 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h"] Oct 30 15:50:47.129575 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:47.129554 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b2a776_bd23_41f3_908d_81cfc68ffb47.slice/crio-13dc7dc5d559c69939072b267344cfe6cf58fd70dbe501501498d25fdd4d7926 WatchSource:0}: Error finding container 13dc7dc5d559c69939072b267344cfe6cf58fd70dbe501501498d25fdd4d7926: Status 404 returned error can't find the container with id 13dc7dc5d559c69939072b267344cfe6cf58fd70dbe501501498d25fdd4d7926 Oct 30 15:50:47.203796 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:47.203772 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9"] Oct 30 15:50:47.204747 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:47.204718 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d911f8e_cab9_4f3d_a1f1_c2a08cd2f407.slice/crio-df383edff83c64c291f75fe6158c70b49c4a041d9f6abc200794bfe1ee0c3ceb WatchSource:0}: Error finding container df383edff83c64c291f75fe6158c70b49c4a041d9f6abc200794bfe1ee0c3ceb: Status 404 returned error can't find the container with id df383edff83c64c291f75fe6158c70b49c4a041d9f6abc200794bfe1ee0c3ceb Oct 30 15:50:47.981169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:47.981139 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78"] Oct 30 15:50:47.985533 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:47.985507 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.003860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.003777 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78"] Oct 30 15:50:48.013617 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.013592 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b63c3c\"" Oct 30 15:50:48.013753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.013637 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b63c3c-dockercfg-5p6md\"" Oct 30 15:50:48.046577 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.046555 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb"] Oct 30 15:50:48.050296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.050277 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.067180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.067155 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb"] Oct 30 15:50:48.095525 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.095490 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" event={"ID":"72b2a776-bd23-41f3-908d-81cfc68ffb47","Type":"ContainerStarted","Data":"8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c"} Oct 30 15:50:48.095662 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.095564 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" event={"ID":"72b2a776-bd23-41f3-908d-81cfc68ffb47","Type":"ContainerStarted","Data":"13dc7dc5d559c69939072b267344cfe6cf58fd70dbe501501498d25fdd4d7926"} Oct 30 15:50:48.095732 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.095685 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:50:48.097311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.097286 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" event={"ID":"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407","Type":"ContainerStarted","Data":"5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e"} Oct 30 15:50:48.097409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.097342 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" event={"ID":"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407","Type":"ContainerStarted","Data":"df383edff83c64c291f75fe6158c70b49c4a041d9f6abc200794bfe1ee0c3ceb"} Oct 30 15:50:48.152808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.151540 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab32413-1c61-458b-87f9-4696ae69aba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.152808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.151596 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4s6g\" (UniqueName: \"kubernetes.io/projected/c28756ca-98c8-47ec-a504-689b12e1052e-kube-api-access-n4s6g\") pod \"payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.152808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.151687 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5p2\" (UniqueName: \"kubernetes.io/projected/9ab32413-1c61-458b-87f9-4696ae69aba9-kube-api-access-ld5p2\") pod \"payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.152808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.151734 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28756ca-98c8-47ec-a504-689b12e1052e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.198412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.198350 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" podStartSLOduration=2.198335159 podStartE2EDuration="2.198335159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:48.15340371 +0000 UTC m=+756.852676870" watchObservedRunningTime="2026-10-30 15:50:48.198335159 +0000 UTC m=+756.897608355" Oct 30 15:50:48.198887 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.198864 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" podStartSLOduration=2.19885495 podStartE2EDuration="2.19885495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:48.197592241 +0000 UTC m=+756.896865398" watchObservedRunningTime="2026-10-30 15:50:48.19885495 +0000 UTC m=+756.898128107" Oct 30 15:50:48.253320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.253252 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab32413-1c61-458b-87f9-4696ae69aba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.253320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.253288 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4s6g\" (UniqueName: \"kubernetes.io/projected/c28756ca-98c8-47ec-a504-689b12e1052e-kube-api-access-n4s6g\") pod \"payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.253524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.253360 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5p2\" (UniqueName: \"kubernetes.io/projected/9ab32413-1c61-458b-87f9-4696ae69aba9-kube-api-access-ld5p2\") pod \"payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.253524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.253406 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28756ca-98c8-47ec-a504-689b12e1052e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.254095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.254072 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab32413-1c61-458b-87f9-4696ae69aba9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.254284 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.254260 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28756ca-98c8-47ec-a504-689b12e1052e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.278716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.278695 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4s6g\" (UniqueName: \"kubernetes.io/projected/c28756ca-98c8-47ec-a504-689b12e1052e-kube-api-access-n4s6g\") pod \"payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.278809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.278773 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5p2\" (UniqueName: \"kubernetes.io/projected/9ab32413-1c61-458b-87f9-4696ae69aba9-kube-api-access-ld5p2\") pod \"payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.300406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.300378 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:48.363035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.363002 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:48.635738 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.635710 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78"] Oct 30 15:50:48.636576 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:48.636551 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28756ca_98c8_47ec_a504_689b12e1052e.slice/crio-f7c16802a7d8dafa45851abb5fcf63b5e0d7cbef2ee2747c94d771a00c923cf3 WatchSource:0}: Error finding container f7c16802a7d8dafa45851abb5fcf63b5e0d7cbef2ee2747c94d771a00c923cf3: Status 404 returned error can't find the container with id f7c16802a7d8dafa45851abb5fcf63b5e0d7cbef2ee2747c94d771a00c923cf3 Oct 30 15:50:48.689783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.689748 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb"] Oct 30 15:50:48.694740 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:48.694708 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab32413_1c61_458b_87f9_4696ae69aba9.slice/crio-55e490b9039adc02f0ca6fd7559677c4d739eb1774bdad56981d091991dd4415 WatchSource:0}: Error finding container 55e490b9039adc02f0ca6fd7559677c4d739eb1774bdad56981d091991dd4415: Status 404 returned error can't find the container with id 55e490b9039adc02f0ca6fd7559677c4d739eb1774bdad56981d091991dd4415 Oct 30 15:50:48.844766 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.844736 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j"] Oct 30 15:50:48.848558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.848541 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:48.868728 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.868701 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-040f9a\"" Oct 30 15:50:48.870691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.870667 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j"] Oct 30 15:50:48.876664 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.876643 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q"] Oct 30 15:50:48.880316 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.880299 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:48.899242 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.899215 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-040f9a-dockercfg-ssdw5\"" Oct 30 15:50:48.904805 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.904779 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q"] Oct 30 15:50:48.961460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.961435 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4txkh\" (UniqueName: \"kubernetes.io/projected/66b3a931-864d-44fe-99d6-813f944c8a3b-kube-api-access-4txkh\") pod \"payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:48.961594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.961478 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b3a931-864d-44fe-99d6-813f944c8a3b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:48.961594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.961509 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbp2\" (UniqueName: \"kubernetes.io/projected/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-kube-api-access-mlbp2\") pod \"payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:48.961594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:48.961574 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:49.062527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.062486 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4txkh\" (UniqueName: \"kubernetes.io/projected/66b3a931-864d-44fe-99d6-813f944c8a3b-kube-api-access-4txkh\") pod \"payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:49.063035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.062554 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b3a931-864d-44fe-99d6-813f944c8a3b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:49.063035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.062592 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbp2\" (UniqueName: \"kubernetes.io/projected/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-kube-api-access-mlbp2\") pod \"payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:49.063035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.062626 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:49.063498 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.063474 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:49.063539 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.063474 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b3a931-864d-44fe-99d6-813f944c8a3b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:49.087757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.087727 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbp2\" (UniqueName: \"kubernetes.io/projected/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-kube-api-access-mlbp2\") pod \"payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:49.087929 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.087770 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4txkh\" (UniqueName: \"kubernetes.io/projected/66b3a931-864d-44fe-99d6-813f944c8a3b-kube-api-access-4txkh\") pod \"payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:49.105974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.105943 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" event={"ID":"c28756ca-98c8-47ec-a504-689b12e1052e","Type":"ContainerStarted","Data":"d1611b097b1bdaea618a5b84be286cbd6b6e1cb117341611b78f3afc02b15311"} Oct 30 15:50:49.106134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.105995 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" event={"ID":"c28756ca-98c8-47ec-a504-689b12e1052e","Type":"ContainerStarted","Data":"f7c16802a7d8dafa45851abb5fcf63b5e0d7cbef2ee2747c94d771a00c923cf3"} Oct 30 15:50:49.106195 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.106129 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:50:49.107580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.107554 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" event={"ID":"9ab32413-1c61-458b-87f9-4696ae69aba9","Type":"ContainerStarted","Data":"98d12d97b494528caf50602dd633980a7544418710a24241a42e618bfd54ab1c"} Oct 30 15:50:49.107699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.107599 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" event={"ID":"9ab32413-1c61-458b-87f9-4696ae69aba9","Type":"ContainerStarted","Data":"55e490b9039adc02f0ca6fd7559677c4d739eb1774bdad56981d091991dd4415"} Oct 30 15:50:49.107885 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.107867 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:50:49.107936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.107912 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:50:49.151752 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.151664 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" podStartSLOduration=2.15164994 podStartE2EDuration="2.15164994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:49.149256557 +0000 UTC m=+757.848529735" watchObservedRunningTime="2026-10-30 15:50:49.15164994 +0000 UTC m=+757.850923112" Oct 30 15:50:49.168726 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.168701 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:49.193998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.193973 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:49.195861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.195796 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" podStartSLOduration=1.195781281 podStartE2EDuration="1.195781281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:49.192534788 +0000 UTC m=+757.891807945" watchObservedRunningTime="2026-10-30 15:50:49.195781281 +0000 UTC m=+757.895054437" Oct 30 15:50:49.478903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.478873 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j"] Oct 30 15:50:49.479851 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:49.479811 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5735b6bb_a75e_4aee_9d7d_7cf796abe26c.slice/crio-8ded4521a521e5c166cb12f222c9698d6603eee80511fd249846968619566f8b WatchSource:0}: Error finding container 8ded4521a521e5c166cb12f222c9698d6603eee80511fd249846968619566f8b: Status 404 returned error can't find the container with id 8ded4521a521e5c166cb12f222c9698d6603eee80511fd249846968619566f8b Oct 30 15:50:49.515314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:49.515285 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q"] Oct 30 15:50:49.516107 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:49.516081 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b3a931_864d_44fe_99d6_813f944c8a3b.slice/crio-b73157fb086ed5a4aa9dc4db0de04c767751de8bfce81221544dfcbedc81b6ff WatchSource:0}: Error finding container b73157fb086ed5a4aa9dc4db0de04c767751de8bfce81221544dfcbedc81b6ff: Status 404 returned error can't find the container with id b73157fb086ed5a4aa9dc4db0de04c767751de8bfce81221544dfcbedc81b6ff Oct 30 15:50:50.114654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.114618 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" event={"ID":"66b3a931-864d-44fe-99d6-813f944c8a3b","Type":"ContainerStarted","Data":"b82affa4b62a9bb0042f60b084677bde3228817daa1dd419e402e28f386fa4d4"} Oct 30 15:50:50.115192 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.114685 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" event={"ID":"66b3a931-864d-44fe-99d6-813f944c8a3b","Type":"ContainerStarted","Data":"b73157fb086ed5a4aa9dc4db0de04c767751de8bfce81221544dfcbedc81b6ff"} Oct 30 15:50:50.115192 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.114811 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:50:50.116304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.116275 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" event={"ID":"5735b6bb-a75e-4aee-9d7d-7cf796abe26c","Type":"ContainerStarted","Data":"e1a96dc874ec5970db728e073417a84c32bf791d371552af821715939ade6932"} Oct 30 15:50:50.116452 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.116318 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" event={"ID":"5735b6bb-a75e-4aee-9d7d-7cf796abe26c","Type":"ContainerStarted","Data":"8ded4521a521e5c166cb12f222c9698d6603eee80511fd249846968619566f8b"} Oct 30 15:50:50.116609 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.116591 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:50:50.167041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.166718 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" podStartSLOduration=2.166695135 podStartE2EDuration="2.166695135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:50.163249011 +0000 UTC m=+758.862522172" watchObservedRunningTime="2026-10-30 15:50:50.166695135 +0000 UTC m=+758.865968292" Oct 30 15:50:50.210897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.210844 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" podStartSLOduration=2.2108103359999998 podStartE2EDuration="2.210810336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:50.208807158 +0000 UTC m=+758.908080316" watchObservedRunningTime="2026-10-30 15:50:50.210810336 +0000 UTC m=+758.910083515" Oct 30 15:50:50.396277 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.396200 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp"] Oct 30 15:50:50.400138 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.400118 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:50.417378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.417337 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-9e2b29-dockercfg-qb7tb\"" Oct 30 15:50:50.423299 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.421153 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp"] Oct 30 15:50:50.479132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.479101 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppxj\" (UniqueName: \"kubernetes.io/projected/4788e294-9ac9-462f-bcad-5fdfdafd2d20-kube-api-access-hppxj\") pod \"payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:50.479332 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.479294 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4788e294-9ac9-462f-bcad-5fdfdafd2d20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:50.508503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.508474 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-9e2b29\"" Oct 30 15:50:50.581098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.581054 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppxj\" (UniqueName: \"kubernetes.io/projected/4788e294-9ac9-462f-bcad-5fdfdafd2d20-kube-api-access-hppxj\") pod \"payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:50.581357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.581336 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4788e294-9ac9-462f-bcad-5fdfdafd2d20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:50.583511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.583492 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4788e294-9ac9-462f-bcad-5fdfdafd2d20-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:50.607897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.607872 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppxj\" (UniqueName: \"kubernetes.io/projected/4788e294-9ac9-462f-bcad-5fdfdafd2d20-kube-api-access-hppxj\") pod \"payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:50.714658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:50.714572 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:51.020984 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.020956 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp"] Oct 30 15:50:51.022077 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:51.022042 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4788e294_9ac9_462f_bcad_5fdfdafd2d20.slice/crio-65eaa254afce31bec701880a526417213f70d4a43bd2254fdf821b7745b9ed44 WatchSource:0}: Error finding container 65eaa254afce31bec701880a526417213f70d4a43bd2254fdf821b7745b9ed44: Status 404 returned error can't find the container with id 65eaa254afce31bec701880a526417213f70d4a43bd2254fdf821b7745b9ed44 Oct 30 15:50:51.124324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.124291 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" event={"ID":"4788e294-9ac9-462f-bcad-5fdfdafd2d20","Type":"ContainerStarted","Data":"7a07aa5051c09b22ff7780ef870534aa1cf317acd5db1c1df64cca1680adf417"} Oct 30 15:50:51.124762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.124356 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" event={"ID":"4788e294-9ac9-462f-bcad-5fdfdafd2d20","Type":"ContainerStarted","Data":"65eaa254afce31bec701880a526417213f70d4a43bd2254fdf821b7745b9ed44"} Oct 30 15:50:51.124762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.124389 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:50:51.168576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.168528 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" podStartSLOduration=1.168515527 podStartE2EDuration="1.168515527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:51.167276972 +0000 UTC m=+759.866550128" watchObservedRunningTime="2026-10-30 15:50:51.168515527 +0000 UTC m=+759.867788730" Oct 30 15:50:51.238025 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.237997 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56"] Oct 30 15:50:51.241680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.241665 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.260619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.260594 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56"] Oct 30 15:50:51.266676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.266655 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a2b3f1-dockercfg-pg2nm\"" Oct 30 15:50:51.319959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.319877 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a2b3f1\"" Oct 30 15:50:51.390705 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.390658 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.390911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.390888 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdc5\" (UniqueName: \"kubernetes.io/projected/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-kube-api-access-kcdc5\") pod \"payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.492300 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.492263 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.492471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.492334 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdc5\" (UniqueName: \"kubernetes.io/projected/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-kube-api-access-kcdc5\") pod \"payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.492989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.492968 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.520200 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.520169 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdc5\" (UniqueName: \"kubernetes.io/projected/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-kube-api-access-kcdc5\") pod \"payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.555179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.555148 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:51.879526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:51.876403 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56"] Oct 30 15:50:51.879526 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:51.877755 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa18168_1e2a_4b93_8a6a_36ab2bcbab7b.slice/crio-14418f5550b213d11bd606095e92dc3a79ed3befcf57740a7211854e44abecad WatchSource:0}: Error finding container 14418f5550b213d11bd606095e92dc3a79ed3befcf57740a7211854e44abecad: Status 404 returned error can't find the container with id 14418f5550b213d11bd606095e92dc3a79ed3befcf57740a7211854e44abecad Oct 30 15:50:51.966563 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:50:51.966531 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:50:52.105368 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:52.105334 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:50:52.131974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:52.131880 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" event={"ID":"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b","Type":"ContainerStarted","Data":"e113cb88be99df19ad677cb27ee659d7e72977caae97ff990d3627db134e729e"} Oct 30 15:50:52.131974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:52.131932 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" event={"ID":"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b","Type":"ContainerStarted","Data":"14418f5550b213d11bd606095e92dc3a79ed3befcf57740a7211854e44abecad"} Oct 30 15:50:52.132391 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:52.131986 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:50:52.207178 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:52.206948 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" podStartSLOduration=1.206927993 podStartE2EDuration="1.206927993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:52.1962291 +0000 UTC m=+760.895502260" watchObservedRunningTime="2026-10-30 15:50:52.206927993 +0000 UTC m=+760.906201161" Oct 30 15:50:53.340430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.340397 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt"] Oct 30 15:50:53.344549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.344532 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:53.373160 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.373114 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt"] Oct 30 15:50:53.401270 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.401242 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-b24e7b\"" Oct 30 15:50:53.401270 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.401240 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-b24e7b-dockercfg-54ktn\"" Oct 30 15:50:53.415653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.415621 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nj7\" (UniqueName: \"kubernetes.io/projected/0c7ea812-ccc6-44d1-88a9-425edea737e5-kube-api-access-s7nj7\") pod \"payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:53.415783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.415656 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7ea812-ccc6-44d1-88a9-425edea737e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:53.522849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.517435 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nj7\" (UniqueName: \"kubernetes.io/projected/0c7ea812-ccc6-44d1-88a9-425edea737e5-kube-api-access-s7nj7\") pod \"payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:53.522849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.517853 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7ea812-ccc6-44d1-88a9-425edea737e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:53.522849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.518740 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7ea812-ccc6-44d1-88a9-425edea737e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:53.572896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.571547 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nj7\" (UniqueName: \"kubernetes.io/projected/0c7ea812-ccc6-44d1-88a9-425edea737e5-kube-api-access-s7nj7\") pod \"payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:53.689908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:53.689876 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:54.001197 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:54.001167 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7ea812_ccc6_44d1_88a9_425edea737e5.slice/crio-9f3f34a50f53e032c48bc361c00f6f6677e8000eee58dd6c936900b1d6f49546 WatchSource:0}: Error finding container 9f3f34a50f53e032c48bc361c00f6f6677e8000eee58dd6c936900b1d6f49546: Status 404 returned error can't find the container with id 9f3f34a50f53e032c48bc361c00f6f6677e8000eee58dd6c936900b1d6f49546 Oct 30 15:50:54.002811 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.002785 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt"] Oct 30 15:50:54.145515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.145479 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" event={"ID":"0c7ea812-ccc6-44d1-88a9-425edea737e5","Type":"ContainerStarted","Data":"3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff"} Oct 30 15:50:54.145706 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.145573 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" event={"ID":"0c7ea812-ccc6-44d1-88a9-425edea737e5","Type":"ContainerStarted","Data":"9f3f34a50f53e032c48bc361c00f6f6677e8000eee58dd6c936900b1d6f49546"} Oct 30 15:50:54.145706 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.145597 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:50:54.190786 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.190751 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw"] Oct 30 15:50:54.194715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.194687 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.197747 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.197701 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" podStartSLOduration=1.19769074 podStartE2EDuration="1.19769074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:54.196503388 +0000 UTC m=+762.895776545" watchObservedRunningTime="2026-10-30 15:50:54.19769074 +0000 UTC m=+762.896963933" Oct 30 15:50:54.215108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.215053 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw"] Oct 30 15:50:54.216921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.216885 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-c129d1-dockercfg-f2spn\"" Oct 30 15:50:54.217078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.217050 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-c129d1\"" Oct 30 15:50:54.333507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.333409 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8475q\" (UniqueName: \"kubernetes.io/projected/28621ed5-d455-427d-bedc-bd96556b40f1-kube-api-access-8475q\") pod \"payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.333739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.333565 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28621ed5-d455-427d-bedc-bd96556b40f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.435070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.435025 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8475q\" (UniqueName: \"kubernetes.io/projected/28621ed5-d455-427d-bedc-bd96556b40f1-kube-api-access-8475q\") pod \"payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.435544 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.435154 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28621ed5-d455-427d-bedc-bd96556b40f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.435875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.435854 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28621ed5-d455-427d-bedc-bd96556b40f1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.470850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.470781 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8475q\" (UniqueName: \"kubernetes.io/projected/28621ed5-d455-427d-bedc-bd96556b40f1-kube-api-access-8475q\") pod \"payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.509342 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.509307 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:54.840518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:54.840482 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw"] Oct 30 15:50:54.841180 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:50:54.841152 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28621ed5_d455_427d_bedc_bd96556b40f1.slice/crio-83c95c749ea66cb49e9ce45b68d458686d70544498e2e12800962cfe0e8126a1 WatchSource:0}: Error finding container 83c95c749ea66cb49e9ce45b68d458686d70544498e2e12800962cfe0e8126a1: Status 404 returned error can't find the container with id 83c95c749ea66cb49e9ce45b68d458686d70544498e2e12800962cfe0e8126a1 Oct 30 15:50:55.176422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:55.176391 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" event={"ID":"28621ed5-d455-427d-bedc-bd96556b40f1","Type":"ContainerStarted","Data":"12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5"} Oct 30 15:50:55.176580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:55.176442 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" event={"ID":"28621ed5-d455-427d-bedc-bd96556b40f1","Type":"ContainerStarted","Data":"83c95c749ea66cb49e9ce45b68d458686d70544498e2e12800962cfe0e8126a1"} Oct 30 15:50:55.176659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:55.176642 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:50:55.222930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:55.222882 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" podStartSLOduration=1.222870163 podStartE2EDuration="1.222870163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:50:55.220514717 +0000 UTC m=+763.919787876" watchObservedRunningTime="2026-10-30 15:50:55.222870163 +0000 UTC m=+763.922143320" Oct 30 15:50:59.110781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:50:59.110750 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 15:51:00.119644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:00.119616 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:51:00.120144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:00.119691 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 15:51:00.120144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:00.119732 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:51:01.127921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:01.127895 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:51:01.302323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:01.302287 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:51:02.135271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:02.135246 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:51:02.135661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:02.135308 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:51:03.141433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:03.141399 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:51:04.277396 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:04.277363 4401 generic.go:356] "Generic (PLEG): container finished" podID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerID="6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5" exitCode=0 Oct 30 15:51:04.277792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:04.277446 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" event={"ID":"25d05b27-3f87-4a35-8502-3ddedaa5456e","Type":"ContainerDied","Data":"6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5"} Oct 30 15:51:04.278044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:04.278026 4401 scope.go:122] "RemoveContainer" containerID="6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5" Oct 30 15:51:04.279699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:04.279664 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:04.279784 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:04.279744 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:04.279833 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:04.279789 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:05.217131 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:05.216080 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:51:05.311472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:05.311437 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" event={"ID":"25d05b27-3f87-4a35-8502-3ddedaa5456e","Type":"ContainerStarted","Data":"07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802"} Oct 30 15:51:05.771848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:05.765227 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:51:05.771848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:05.765359 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 15:51:05.771848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:05.765404 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 15:51:06.197408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:06.196383 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:51:06.237637 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:06.237584 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:51:06.758694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:06.758226 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 15:51:06.758694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:06.758301 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 15:51:07.766658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:07.766203 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 15:51:07.766658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:07.766265 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 15:51:08.333634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.333538 4401 generic.go:356] "Generic (PLEG): container finished" podID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerID="cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988" exitCode=0 Oct 30 15:51:08.333634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.333603 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" event={"ID":"564bbe1a-6e89-4b9f-9432-795b6205f0a8","Type":"ContainerDied","Data":"cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988"} Oct 30 15:51:08.334049 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.334032 4401 scope.go:122] "RemoveContainer" containerID="cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988" Oct 30 15:51:08.335630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.335598 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:08.335734 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.335682 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:08.335798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.335730 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:08.764851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.756679 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 15:51:08.764851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:08.756743 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 15:51:09.342255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:09.342058 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" event={"ID":"564bbe1a-6e89-4b9f-9432-795b6205f0a8","Type":"ContainerStarted","Data":"50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d"} Oct 30 15:51:09.388228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:09.388195 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:51:09.396039 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:09.396003 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 15:51:09.396313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:09.396286 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 15:51:09.759681 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:09.759234 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 15:51:09.759681 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:09.759302 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 15:51:10.393909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.384226 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 15:51:10.393909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.384289 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 15:51:10.726022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.725928 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7"] Oct 30 15:51:10.730911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.730883 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.755879 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.755808 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b63c3c-openshift-default-dockercfg-94q4v\"" Oct 30 15:51:10.771846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.769199 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7"] Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.813707 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.813771 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.813805 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m9x8\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-kube-api-access-8m9x8\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.813957 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.814011 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.814037 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.814107 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.814135 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.814392 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.814220 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.886792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.886755 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:51:10.887227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.887207 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:51:10.888318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.888288 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:51:10.915144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915111 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915321 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915224 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915407 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915385 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915469 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915429 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m9x8\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-kube-api-access-8m9x8\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915525 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915572 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915592 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915622 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915707 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915739 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915852 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.915990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915527 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.916052 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.915993 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.916157 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.916138 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.918026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.917992 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.918141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.918101 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.942968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.942939 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m9x8\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-kube-api-access-8m9x8\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:10.943350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:10.943331 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:11.072781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.063085 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:11.092303 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.092259 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm"] Oct 30 15:51:11.131995 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.131966 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.135647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.135589 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm"] Oct 30 15:51:11.221549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221513 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221563 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221590 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221612 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94tw6\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-kube-api-access-94tw6\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221647 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221693 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4138b-2339-4c0f-a8f7-31d535aec102-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221735 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221768 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.221944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.221863 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323463 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323518 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323543 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323568 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94tw6\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-kube-api-access-94tw6\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323602 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323645 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4138b-2339-4c0f-a8f7-31d535aec102-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323683 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323716 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.323922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323789 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.324238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.323996 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.324294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.324273 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.325420 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.325144 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4138b-2339-4c0f-a8f7-31d535aec102-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.326385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.325661 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.333036 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.330977 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.333036 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.331011 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.333036 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.331464 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.375353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.375326 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94tw6\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-kube-api-access-94tw6\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.375863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.375818 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.383968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.383940 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 15:51:11.384081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.383980 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 15:51:11.472586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.472547 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7"] Oct 30 15:51:11.480676 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:51:11.480649 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c97ffa2_d41e_4790_b6dc_34f809bf05f2.slice/crio-7bd7b57489ff9e140ce114fc50fa52d77fbcb40c0c3df52f0221e62f7e2d53b7 WatchSource:0}: Error finding container 7bd7b57489ff9e140ce114fc50fa52d77fbcb40c0c3df52f0221e62f7e2d53b7: Status 404 returned error can't find the container with id 7bd7b57489ff9e140ce114fc50fa52d77fbcb40c0c3df52f0221e62f7e2d53b7 Oct 30 15:51:11.482322 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.482296 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:11.484015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.483973 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:11.484089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.484065 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:11.484128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.484109 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:11.875538 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.875506 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm"] Oct 30 15:51:11.878062 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:51:11.878034 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda4138b_2339_4c0f_a8f7_31d535aec102.slice/crio-f5690e4ad06fb1f8dc3e9ed8326490efb72ca34b03f67ab5c48bc31704c7e71e WatchSource:0}: Error finding container f5690e4ad06fb1f8dc3e9ed8326490efb72ca34b03f67ab5c48bc31704c7e71e: Status 404 returned error can't find the container with id f5690e4ad06fb1f8dc3e9ed8326490efb72ca34b03f67ab5c48bc31704c7e71e Oct 30 15:51:11.881000 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.880966 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:11.881110 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.881048 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:11.881110 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:11.881094 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:12.404344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.404278 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 15:51:12.404550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.404343 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 15:51:12.415777 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.415729 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" event={"ID":"dda4138b-2339-4c0f-a8f7-31d535aec102","Type":"ContainerStarted","Data":"ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3"} Oct 30 15:51:12.415982 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.415809 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" event={"ID":"dda4138b-2339-4c0f-a8f7-31d535aec102","Type":"ContainerStarted","Data":"f5690e4ad06fb1f8dc3e9ed8326490efb72ca34b03f67ab5c48bc31704c7e71e"} Oct 30 15:51:12.419938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.419865 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" event={"ID":"4c97ffa2-d41e-4790-b6dc-34f809bf05f2","Type":"ContainerStarted","Data":"48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd"} Oct 30 15:51:12.420095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.419954 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" event={"ID":"4c97ffa2-d41e-4790-b6dc-34f809bf05f2","Type":"ContainerStarted","Data":"7bd7b57489ff9e140ce114fc50fa52d77fbcb40c0c3df52f0221e62f7e2d53b7"} Oct 30 15:51:12.473468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.473410 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podStartSLOduration=1.473391491 podStartE2EDuration="1.473391491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:12.471605725 +0000 UTC m=+781.170878909" watchObservedRunningTime="2026-10-30 15:51:12.473391491 +0000 UTC m=+781.172664650" Oct 30 15:51:12.488182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.488148 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:12.489515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.489483 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:51:12.490735 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.490706 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:51:12.525199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:12.525131 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" podStartSLOduration=2.525109861 podStartE2EDuration="2.525109861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:12.520971353 +0000 UTC m=+781.220244578" watchObservedRunningTime="2026-10-30 15:51:12.525109861 +0000 UTC m=+781.224383019" Oct 30 15:51:13.071852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.071018 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:51:13.073673 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.073614 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:13.096812 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.096776 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:51:13.097125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.097093 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:51:13.391683 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.391237 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 15:51:13.391683 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.391301 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 15:51:13.515860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.499972 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:51:13.515860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:13.500051 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:51:14.076849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.073426 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:51:14.076849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.073489 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:51:14.116851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.116629 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:51:14.116851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.116693 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:51:14.386853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.386250 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Oct 30 15:51:14.386853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.386314 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Oct 30 15:51:14.492851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.483324 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:51:14.492851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:14.483403 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:51:15.080849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.079267 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:51:15.080849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.079332 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:51:15.399881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.399391 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:51:15.401077 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.401054 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:51:15.445194 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.445158 4401 generic.go:356] "Generic (PLEG): container finished" podID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerID="beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b" exitCode=0 Oct 30 15:51:15.445453 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.445432 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" event={"ID":"fb734159-1499-4633-9a9e-aefdfcf9b76f","Type":"ContainerDied","Data":"beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b"} Oct 30 15:51:15.445738 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.445724 4401 scope.go:122] "RemoveContainer" containerID="beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b" Oct 30 15:51:15.466716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.466669 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:15.467001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.466976 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:15.467160 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.467141 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:15.507883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.498217 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:51:15.507883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:15.498275 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:51:16.077908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:16.077195 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:51:16.077908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:16.077260 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:51:16.487752 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:16.487715 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:51:16.491618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:16.491587 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:51:16.492241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:16.490571 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" event={"ID":"fb734159-1499-4633-9a9e-aefdfcf9b76f","Type":"ContainerStarted","Data":"48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49"} Oct 30 15:51:17.080847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:17.076266 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 15:51:17.080847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:17.076333 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 15:51:17.219293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:17.219254 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:51:17.219748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:17.219723 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 15:51:17.219998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:17.219970 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 15:51:17.483265 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:17.483222 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:51:17.483540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:17.483512 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:51:18.220078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:18.220044 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 15:51:18.220538 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:18.220514 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 15:51:18.471200 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:18.471118 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:18.472849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:18.472811 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:18.482084 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:18.482057 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:51:18.483399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:18.483370 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:51:18.483518 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:18.483431 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 15:51:19.032572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:19.032534 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:51:19.219425 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:19.219389 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 15:51:19.219593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:19.219443 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 15:51:19.494689 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:19.494649 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:19.497073 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:19.497044 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:19.500612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:19.500584 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:51:20.219060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.219029 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 15:51:20.219229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.219079 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 15:51:20.488816 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.488739 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985"] Oct 30 15:51:20.493116 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.493092 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.510027 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.510004 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985"] Oct 30 15:51:20.520931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.520906 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9e2b29\"/\"simulator-w2-9e2b29-kserve-self-signed-certs\"" Oct 30 15:51:20.521095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.520961 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9e2b29\"/\"default-dockercfg-8ttrr\"" Oct 30 15:51:20.525901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.525879 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9e2b29\"/\"openshift-service-ca.crt\"" Oct 30 15:51:20.525983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.525921 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-9e2b29\"/\"kube-root-ca.crt\"" Oct 30 15:51:20.679746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.679707 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-home\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.679973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.679781 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-dshm\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.679973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.679856 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxd5\" (UniqueName: \"kubernetes.io/projected/a2f5c3db-a820-403d-a478-91c0ebe886c0-kube-api-access-lcxd5\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.679973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.679921 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f5c3db-a820-403d-a478-91c0ebe886c0-tls-certs\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.679973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.679952 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-tmp-dir\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.680147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.680079 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-model-cache\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.781565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781483 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-home\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.781565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781535 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-dshm\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.781802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781588 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxd5\" (UniqueName: \"kubernetes.io/projected/a2f5c3db-a820-403d-a478-91c0ebe886c0-kube-api-access-lcxd5\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.781802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781637 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f5c3db-a820-403d-a478-91c0ebe886c0-tls-certs\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.781802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781667 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-tmp-dir\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.781802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781701 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-model-cache\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.782061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781961 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-home\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.782061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.781986 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-tmp-dir\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.782061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.782058 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-model-cache\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.783920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.783902 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-dshm\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.784088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.784072 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f5c3db-a820-403d-a478-91c0ebe886c0-tls-certs\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.806479 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.806450 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxd5\" (UniqueName: \"kubernetes.io/projected/a2f5c3db-a820-403d-a478-91c0ebe886c0-kube-api-access-lcxd5\") pod \"simulator-w2-9e2b29-kserve-bbd645884-bf985\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:20.807262 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:20.807243 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:21.219200 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:21.219167 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 15:51:21.219493 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:21.219466 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 15:51:21.229339 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:21.229311 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985"] Oct 30 15:51:21.230094 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:51:21.230066 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2f5c3db_a820_403d_a478_91c0ebe886c0.slice/crio-ab51106ee1322a65b670d52a1e5b750db62c8eebe102cfbdfa141efcf4617310 WatchSource:0}: Error finding container ab51106ee1322a65b670d52a1e5b750db62c8eebe102cfbdfa141efcf4617310: Status 404 returned error can't find the container with id ab51106ee1322a65b670d52a1e5b750db62c8eebe102cfbdfa141efcf4617310 Oct 30 15:51:21.591585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:21.591486 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" event={"ID":"a2f5c3db-a820-403d-a478-91c0ebe886c0","Type":"ContainerStarted","Data":"9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566"} Oct 30 15:51:21.591585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:21.591559 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" event={"ID":"a2f5c3db-a820-403d-a478-91c0ebe886c0","Type":"ContainerStarted","Data":"ab51106ee1322a65b670d52a1e5b750db62c8eebe102cfbdfa141efcf4617310"} Oct 30 15:51:21.636756 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:21.636699 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podStartSLOduration=1.636680003 podStartE2EDuration="1.636680003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:21.635114954 +0000 UTC m=+790.334388113" watchObservedRunningTime="2026-10-30 15:51:21.636680003 +0000 UTC m=+790.335953162" Oct 30 15:51:21.807544 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:21.807514 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:21.967625 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:51:21.967580 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:51:22.223132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:22.223062 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:51:22.223354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:22.223336 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:51:22.224194 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:22.224157 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:51:30.807946 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:30.807908 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:30.820633 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:30.820606 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:31.681508 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:31.681481 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:51:33.188002 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.187970 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf"] Oct 30 15:51:33.191516 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.191500 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.210410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.210374 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf"] Oct 30 15:51:33.221434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.221409 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a2b3f1\"/\"openshift-service-ca.crt\"" Oct 30 15:51:33.221589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.221438 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a2b3f1\"/\"kube-root-ca.crt\"" Oct 30 15:51:33.221589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.221497 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2b3f1\"/\"simulator-w1-a2b3f1-kserve-self-signed-certs\"" Oct 30 15:51:33.222005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.221989 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2b3f1\"/\"default-dockercfg-gvgcj\"" Oct 30 15:51:33.234274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.234253 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-dshm\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.234379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.234285 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf71f9-764f-4251-bf81-bb3222b85808-tls-certs\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.234379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.234322 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-model-cache\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.234515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.234493 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-home\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.234580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.234538 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-tmp-dir\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.234667 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.234637 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t99gj\" (UniqueName: \"kubernetes.io/projected/6dcf71f9-764f-4251-bf81-bb3222b85808-kube-api-access-t99gj\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.335384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.335355 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t99gj\" (UniqueName: \"kubernetes.io/projected/6dcf71f9-764f-4251-bf81-bb3222b85808-kube-api-access-t99gj\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.335549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.335502 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-dshm\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.335549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.335533 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf71f9-764f-4251-bf81-bb3222b85808-tls-certs\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.335653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.335598 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-model-cache\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.335653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.335650 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-home\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.335859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.335802 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-tmp-dir\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.336088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.336065 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-model-cache\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.336176 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.336089 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-home\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.336249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.336232 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-tmp-dir\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.338359 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.338336 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-dshm\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.338785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.338766 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf71f9-764f-4251-bf81-bb3222b85808-tls-certs\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.362137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.362117 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t99gj\" (UniqueName: \"kubernetes.io/projected/6dcf71f9-764f-4251-bf81-bb3222b85808-kube-api-access-t99gj\") pod \"simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.504286 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.504212 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:33.812408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:33.812376 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf"] Oct 30 15:51:33.812766 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:51:33.812741 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dcf71f9_764f_4251_bf81_bb3222b85808.slice/crio-ff6ae24ab1e4ee42ead0fdf95165d21750920c2522ceb5f67a1ad6cf822af9a0 WatchSource:0}: Error finding container ff6ae24ab1e4ee42ead0fdf95165d21750920c2522ceb5f67a1ad6cf822af9a0: Status 404 returned error can't find the container with id ff6ae24ab1e4ee42ead0fdf95165d21750920c2522ceb5f67a1ad6cf822af9a0 Oct 30 15:51:34.686127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:34.686093 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" event={"ID":"6dcf71f9-764f-4251-bf81-bb3222b85808","Type":"ContainerStarted","Data":"5c94bba59ffdad9cb6813c726f92ab8b145d560b9073376f0c0cad68d8363e91"} Oct 30 15:51:34.686537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:34.686136 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" event={"ID":"6dcf71f9-764f-4251-bf81-bb3222b85808","Type":"ContainerStarted","Data":"ff6ae24ab1e4ee42ead0fdf95165d21750920c2522ceb5f67a1ad6cf822af9a0"} Oct 30 15:51:34.733158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:34.733108 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podStartSLOduration=1.7330952370000001 podStartE2EDuration="1.733095237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:34.730862096 +0000 UTC m=+803.430135250" watchObservedRunningTime="2026-10-30 15:51:34.733095237 +0000 UTC m=+803.432368393" Oct 30 15:51:35.505058 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.505023 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:35.698998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.698965 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt"] Oct 30 15:51:35.703326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.703303 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.725850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.725793 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt"] Oct 30 15:51:35.736470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.736447 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b24e7b\"/\"openshift-service-ca.crt\"" Oct 30 15:51:35.736604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.736510 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b24e7b\"/\"kube-root-ca.crt\"" Oct 30 15:51:35.738697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.738682 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b24e7b\"/\"default-dockercfg-h2s6c\"" Oct 30 15:51:35.739145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.739129 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b24e7b\"/\"simulator-w4-b24e7b-kserve-self-signed-certs\"" Oct 30 15:51:35.769643 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.769562 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-home\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.769643 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.769598 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-tmp-dir\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.769884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.769790 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb8gm\" (UniqueName: \"kubernetes.io/projected/8e596300-e655-4aeb-ac47-feb28b3463dc-kube-api-access-hb8gm\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.769969 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.769954 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-dshm\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.770033 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.770021 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-model-cache\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.770083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.770056 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e596300-e655-4aeb-ac47-feb28b3463dc-tls-certs\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.870792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.870757 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-tmp-dir\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.870808 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb8gm\" (UniqueName: \"kubernetes.io/projected/8e596300-e655-4aeb-ac47-feb28b3463dc-kube-api-access-hb8gm\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.870896 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-dshm\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.870936 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-model-cache\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.870957 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e596300-e655-4aeb-ac47-feb28b3463dc-tls-certs\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871279 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.871038 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-home\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871279 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.871208 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-tmp-dir\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871374 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.871353 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-model-cache\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.871423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.871392 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-home\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.873135 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.873115 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-dshm\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.873470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.873454 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e596300-e655-4aeb-ac47-feb28b3463dc-tls-certs\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:35.895775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:35.895746 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb8gm\" (UniqueName: \"kubernetes.io/projected/8e596300-e655-4aeb-ac47-feb28b3463dc-kube-api-access-hb8gm\") pod \"simulator-w4-b24e7b-kserve-54d647b657-g65lt\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:36.018773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:36.018733 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:36.324070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:36.324041 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt"] Oct 30 15:51:36.325050 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:51:36.325025 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e596300_e655_4aeb_ac47_feb28b3463dc.slice/crio-e7f220dbfc4189c08846d63172d6c208589e62d69421b4e709ab9e60131e08b8 WatchSource:0}: Error finding container e7f220dbfc4189c08846d63172d6c208589e62d69421b4e709ab9e60131e08b8: Status 404 returned error can't find the container with id e7f220dbfc4189c08846d63172d6c208589e62d69421b4e709ab9e60131e08b8 Oct 30 15:51:36.509924 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:36.509888 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:51:36.703440 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:36.703393 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" event={"ID":"8e596300-e655-4aeb-ac47-feb28b3463dc","Type":"ContainerStarted","Data":"03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91"} Oct 30 15:51:36.703847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:36.703446 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" event={"ID":"8e596300-e655-4aeb-ac47-feb28b3463dc","Type":"ContainerStarted","Data":"e7f220dbfc4189c08846d63172d6c208589e62d69421b4e709ab9e60131e08b8"} Oct 30 15:51:36.749727 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:36.749686 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podStartSLOduration=1.74967385 podStartE2EDuration="1.74967385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:36.747739913 +0000 UTC m=+805.447013070" watchObservedRunningTime="2026-10-30 15:51:36.74967385 +0000 UTC m=+805.448947023" Oct 30 15:51:37.019483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:37.019402 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:40.802131 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.802070 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6"] Oct 30 15:51:40.806620 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.806600 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828112 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828079 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828146 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828251 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828294 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls95n\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-kube-api-access-ls95n\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828315 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828383 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828369 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828518 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828563 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.828751 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.828709 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.829117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.829091 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6"] Oct 30 15:51:40.844586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.844563 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6"] Oct 30 15:51:40.848542 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.848523 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.865450 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.865428 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6"] Oct 30 15:51:40.929540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929505 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.929676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929548 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.929676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929574 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.929676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929619 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnmb\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-kube-api-access-gcnmb\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.929676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929645 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.929676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929660 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.929913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929755 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.929913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929788 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.929913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929809 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls95n\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-kube-api-access-ls95n\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.929913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929860 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929906 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.929979 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.930099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930003 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930037 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930065 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930099 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930133 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.930366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930163 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.930366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930190 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930200 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930269 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:40.930578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930552 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.930775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.930753 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.931805 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.931783 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.932311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.932296 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.957558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.957522 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.957705 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.957621 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls95n\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-kube-api-access-ls95n\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:40.965594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.965567 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c"] Oct 30 15:51:40.971200 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.971181 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:40.988934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.988907 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ba9db9-openshift-default-dockercfg-lczk5\"" Oct 30 15:51:40.992298 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:40.992277 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c"] Oct 30 15:51:41.031580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031550 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.031580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031584 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.031781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031606 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnmb\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-kube-api-access-gcnmb\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.031781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031666 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.031781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031703 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.031781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031749 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031927 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.032011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031957 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a807894c-cb95-462f-92da-61cb4a059e95-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.032011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.031973 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a807894c-cb95-462f-92da-61cb4a059e95-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.032170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032087 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032225 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032207 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032283 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032240 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-credential-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032283 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032244 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2zdj\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-kube-api-access-v2zdj\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.032463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032327 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.032463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032363 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.032463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032392 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-certs\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032409 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032472 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032510 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032553 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.032848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032680 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-data\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.032848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.032510 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-socket\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.033168 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.033147 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istiod-ca-cert\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.034106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.034083 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-envoy\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.034239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.034221 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-podinfo\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.059958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.059890 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnmb\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-kube-api-access-gcnmb\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.060222 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.060202 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-token\") pod \"e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.121059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.121026 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:41.133489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133465 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133527 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133555 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133576 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133605 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a807894c-cb95-462f-92da-61cb4a059e95-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133630 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a807894c-cb95-462f-92da-61cb4a059e95-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133704 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2zdj\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-kube-api-access-v2zdj\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133739 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.133843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133769 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.134202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.133897 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.134202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.134038 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.134202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.134122 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.134333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.134259 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.134708 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.134682 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a807894c-cb95-462f-92da-61cb4a059e95-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.136162 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.136140 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a807894c-cb95-462f-92da-61cb4a059e95-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.136385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.136368 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.160770 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.160743 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2zdj\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-kube-api-access-v2zdj\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.165528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.165504 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.166338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.166313 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:41.201186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.201142 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd"] Oct 30 15:51:41.209731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.209160 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.222849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.221845 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd"] Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235684 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235741 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235777 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235791 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be71f92e-9b24-4e70-911d-a6b62e659d56-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235864 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235890 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235907 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cldp8\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-kube-api-access-cldp8\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235948 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.237296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.235978 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.288119 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.288087 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:41.337619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.337532 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.337972 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.338443 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be71f92e-9b24-4e70-911d-a6b62e659d56-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.338546 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.338879 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.338923 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cldp8\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-kube-api-access-cldp8\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.338933 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-data\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.339003 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.339052 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.339129 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.339314 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be71f92e-9b24-4e70-911d-a6b62e659d56-istiod-ca-cert\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.339425 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-certs\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.339911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.339693 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.341457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.341415 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-envoy\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.342583 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.342339 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-credential-socket\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.342583 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.342545 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-podinfo\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.362867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.362841 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-token\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.363171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.363153 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cldp8\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-kube-api-access-cldp8\") pod \"e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.446679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.446648 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6"] Oct 30 15:51:41.446811 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:51:41.446715 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d62b438_6f71_4ea0_9493_3ce8f382fe21.slice/crio-36107c4295ec958822fa7c86158482d9993365c3533569f8c85672b2f6d2d53d WatchSource:0}: Error finding container 36107c4295ec958822fa7c86158482d9993365c3533569f8c85672b2f6d2d53d: Status 404 returned error can't find the container with id 36107c4295ec958822fa7c86158482d9993365c3533569f8c85672b2f6d2d53d Oct 30 15:51:41.449024 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.448996 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.449094 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.449054 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.449094 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.449083 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.498039 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.498012 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6"] Oct 30 15:51:41.499160 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:51:41.499138 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c0f1fd_4e1c_4f99_a59d_8e5d711a1a84.slice/crio-df9f4647498a4323d5414af5b39a6128ee1c26f17009392337f5f77a2092f5a7 WatchSource:0}: Error finding container df9f4647498a4323d5414af5b39a6128ee1c26f17009392337f5f77a2092f5a7: Status 404 returned error can't find the container with id df9f4647498a4323d5414af5b39a6128ee1c26f17009392337f5f77a2092f5a7 Oct 30 15:51:41.502077 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.502045 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.502151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.502108 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.502151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.502136 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.534608 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.534582 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:41.609562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.603062 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c"] Oct 30 15:51:41.617693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.617172 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.617693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.617273 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.617693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.617319 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.745133 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.745102 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" event={"ID":"a807894c-cb95-462f-92da-61cb4a059e95","Type":"ContainerStarted","Data":"5f0c6fb3d6149c8eaa4e08dc7e8dd3be365e49859601ad32b6e21285a34b3239"} Oct 30 15:51:41.749193 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.749167 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" event={"ID":"4d62b438-6f71-4ea0-9493-3ce8f382fe21","Type":"ContainerStarted","Data":"4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4"} Oct 30 15:51:41.749360 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.749342 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" event={"ID":"4d62b438-6f71-4ea0-9493-3ce8f382fe21","Type":"ContainerStarted","Data":"36107c4295ec958822fa7c86158482d9993365c3533569f8c85672b2f6d2d53d"} Oct 30 15:51:41.751662 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.751639 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" event={"ID":"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84","Type":"ContainerStarted","Data":"81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd"} Oct 30 15:51:41.751778 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.751679 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" event={"ID":"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84","Type":"ContainerStarted","Data":"df9f4647498a4323d5414af5b39a6128ee1c26f17009392337f5f77a2092f5a7"} Oct 30 15:51:41.828439 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.828363 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podStartSLOduration=1.828322504 podStartE2EDuration="1.828322504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:41.802292625 +0000 UTC m=+810.501565783" watchObservedRunningTime="2026-10-30 15:51:41.828322504 +0000 UTC m=+810.527595661" Oct 30 15:51:41.871042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.870977 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podStartSLOduration=1.870951116 podStartE2EDuration="1.870951116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:41.859450437 +0000 UTC m=+810.558723594" watchObservedRunningTime="2026-10-30 15:51:41.870951116 +0000 UTC m=+810.570224274" Oct 30 15:51:41.892848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.892674 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.893599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.892819 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.893599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.893080 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:51:41.898532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:41.898510 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd"] Oct 30 15:51:42.122314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.122235 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:42.123706 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.123676 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:51:42.123855 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.123749 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:51:42.176983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.176952 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:42.180852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.177902 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:51:42.180852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.177955 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:51:42.812913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.811772 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" event={"ID":"be71f92e-9b24-4e70-911d-a6b62e659d56","Type":"ContainerStarted","Data":"637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd"} Oct 30 15:51:42.812913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.811857 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" event={"ID":"be71f92e-9b24-4e70-911d-a6b62e659d56","Type":"ContainerStarted","Data":"79471d652f5c06fee489d32dbde67f358cb8b67ae0b2e03556d11565d881c0a9"} Oct 30 15:51:42.859097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.859062 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" event={"ID":"a807894c-cb95-462f-92da-61cb4a059e95","Type":"ContainerStarted","Data":"1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121"} Oct 30 15:51:42.899899 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.899813 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podStartSLOduration=1.899792744 podStartE2EDuration="1.899792744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:42.869257485 +0000 UTC m=+811.568530643" watchObservedRunningTime="2026-10-30 15:51:42.899792744 +0000 UTC m=+811.599065901" Oct 30 15:51:42.956086 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:42.956015 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podStartSLOduration=2.9559909060000003 podStartE2EDuration="2.955990906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:42.937322504 +0000 UTC m=+811.636595662" watchObservedRunningTime="2026-10-30 15:51:42.955990906 +0000 UTC m=+811.655264064" Oct 30 15:51:43.122268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.122227 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:51:43.122565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.122535 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:51:43.176565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.176528 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:51:43.176876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.176842 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:51:43.291849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.290986 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:43.292979 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.292947 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:51:43.293189 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.293165 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:51:43.508919 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.507857 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:43.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.536631 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:43.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.536781 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:51:43.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.536856 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:51:43.837306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:43.837218 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:44.122845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.122233 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:51:44.122845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.122294 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:51:44.173848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.173772 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:51:44.174049 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.173879 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:51:44.296851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.295995 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:51:44.296851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.296058 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:51:44.405902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.405164 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:51:44.653903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.653858 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:51:44.668893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:44.654283 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:51:45.122290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.122251 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:51:45.122598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.122559 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:51:45.173847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.173310 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:51:45.173847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.173387 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:51:45.290864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.290190 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:51:45.290864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.290251 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:51:45.536354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.536258 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:51:45.536623 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:45.536590 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:51:46.022055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.021844 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:46.123964 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.123906 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:46.124757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.124728 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:51:46.124998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.124972 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:51:46.182003 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.181961 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:51:46.182620 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.182583 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:51:46.295916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.295165 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:51:46.295916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.295239 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:51:46.538249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.538212 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:51:46.538507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:46.538478 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:51:47.121795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.121758 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:51:47.122110 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.122078 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:51:47.169225 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.169185 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:51:47.169475 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.169447 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:51:47.247736 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.247704 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:51:47.308500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.308469 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:51:47.308803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.308756 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:51:47.589706 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.588930 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:51:47.589706 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:47.588991 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:51:48.128290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.128250 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:51:48.128691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.128657 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Oct 30 15:51:48.171849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.168216 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:51:48.171849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.168274 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 30 15:51:48.289315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.289277 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 30 15:51:48.289905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.289872 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 30 15:51:48.535398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.535179 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 15:51:48.535398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:48.535239 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 15:51:49.380978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.380728 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:49.382274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.382231 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:49.383124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.383105 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:51:49.415110 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.415081 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:49.416449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.416195 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:49.417694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.416962 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 15:51:49.434884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.434357 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:49.436545 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.436519 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:49.436691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.436612 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:51:49.780091 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.780012 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:49.780684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.780655 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:49.781554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:49.781532 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:51:51.966476 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:51:51.966443 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:51:52.718846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:51:52.718389 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:52:05.482106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:05.482074 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:52:07.527329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.527292 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj"] Oct 30 15:52:07.531676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.531652 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.547306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.547280 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj"] Oct 30 15:52:07.558210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.558182 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b63c3c\"/\"premium-w0-b63c3c-kserve-self-signed-certs\"" Oct 30 15:52:07.558356 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.558234 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b63c3c\"/\"default-dockercfg-wksm9\"" Oct 30 15:52:07.562313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.562284 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b63c3c\"/\"openshift-service-ca.crt\"" Oct 30 15:52:07.562405 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.562365 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b63c3c\"/\"kube-root-ca.crt\"" Oct 30 15:52:07.629679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.629650 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd7982-3c33-4b37-9f09-336d50de082d-tls-certs\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.629859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.629692 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-dshm\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.629859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.629714 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-home\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.629859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.629789 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwjgk\" (UniqueName: \"kubernetes.io/projected/a8bd7982-3c33-4b37-9f09-336d50de082d-kube-api-access-pwjgk\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.629982 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.629890 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-tmp-dir\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.630018 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.630003 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-model-cache\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.738514 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-model-cache\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.738615 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd7982-3c33-4b37-9f09-336d50de082d-tls-certs\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.738662 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-dshm\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.738689 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-home\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.738735 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwjgk\" (UniqueName: \"kubernetes.io/projected/a8bd7982-3c33-4b37-9f09-336d50de082d-kube-api-access-pwjgk\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.738819 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-tmp-dir\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.739351 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-tmp-dir\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.741850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.739648 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-model-cache\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.756869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.749435 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-home\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.771847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.770670 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd7982-3c33-4b37-9f09-336d50de082d-tls-certs\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.799816 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.799159 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-dshm\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.814130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.811073 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwjgk\" (UniqueName: \"kubernetes.io/projected/a8bd7982-3c33-4b37-9f09-336d50de082d-kube-api-access-pwjgk\") pod \"premium-w0-b63c3c-kserve-758657f9cd-hhnmj\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:07.844793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:07.844273 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:08.155783 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:52:08.155754 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8bd7982_3c33_4b37_9f09_336d50de082d.slice/crio-e73ecbf25eac3ac3c4c76b3c706d73c27ea5cb2131f62a5309b36b7c726c492d WatchSource:0}: Error finding container e73ecbf25eac3ac3c4c76b3c706d73c27ea5cb2131f62a5309b36b7c726c492d: Status 404 returned error can't find the container with id e73ecbf25eac3ac3c4c76b3c706d73c27ea5cb2131f62a5309b36b7c726c492d Oct 30 15:52:08.157675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:08.157648 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj"] Oct 30 15:52:08.244082 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:08.244047 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" event={"ID":"a8bd7982-3c33-4b37-9f09-336d50de082d","Type":"ContainerStarted","Data":"8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0"} Oct 30 15:52:08.244231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:08.244098 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" event={"ID":"a8bd7982-3c33-4b37-9f09-336d50de082d","Type":"ContainerStarted","Data":"e73ecbf25eac3ac3c4c76b3c706d73c27ea5cb2131f62a5309b36b7c726c492d"} Oct 30 15:52:08.290476 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:08.290428 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podStartSLOduration=1.290414974 podStartE2EDuration="1.290414974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:08.288689741 +0000 UTC m=+836.987962898" watchObservedRunningTime="2026-10-30 15:52:08.290414974 +0000 UTC m=+836.989688170" Oct 30 15:52:08.844637 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:08.844605 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:11.601897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.601865 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb"] Oct 30 15:52:11.606999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.606978 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.639945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.639913 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-c129d1-openshift-default-dockercfg-qvk6s\"" Oct 30 15:52:11.645804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.645779 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb"] Oct 30 15:52:11.703984 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.703954 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.703984 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.703986 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.704180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.704050 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.704180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.704129 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.704180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.704164 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.704269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.704232 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.704333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.704321 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.704397 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.704380 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.704433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.704416 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28qw5\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-kube-api-access-28qw5\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.805716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805683 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.805716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805718 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.805973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805739 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28qw5\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-kube-api-access-28qw5\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.805973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805835 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.805973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805864 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.805973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805908 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.805973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805944 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.806177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.805975 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.806177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.806021 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.806250 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.806190 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.806542 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.806518 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.806648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.806537 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.806648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.806555 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.806717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.806683 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.808307 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.808284 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.808429 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.808342 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.833422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.833400 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.833589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.833571 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28qw5\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-kube-api-access-28qw5\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:11.920692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:11.920597 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:12.291031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:12.290982 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb"] Oct 30 15:52:12.301291 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:12.301250 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:52:12.301423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:12.301337 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:52:12.301423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:12.301382 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:52:13.328880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:13.328256 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" event={"ID":"e64b4190-cdf8-45e6-83b5-aa6081a311f0","Type":"ContainerStarted","Data":"2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935"} Oct 30 15:52:13.328880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:13.328329 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" event={"ID":"e64b4190-cdf8-45e6-83b5-aa6081a311f0","Type":"ContainerStarted","Data":"97314904364bbcce9ba5297955b0e127daeef5ea6c63dbfc8fc0e3eab6286b4f"} Oct 30 15:52:13.429851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:13.429577 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podStartSLOduration=2.429556535 podStartE2EDuration="2.429556535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:13.407282458 +0000 UTC m=+842.106555616" watchObservedRunningTime="2026-10-30 15:52:13.429556535 +0000 UTC m=+842.128829693" Oct 30 15:52:13.921034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:13.920999 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:13.922783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:13.922756 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:52:13.922942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:13.922813 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:52:14.921408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:14.921374 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:52:14.921921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:14.921433 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:52:15.921524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:15.921490 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:52:15.922001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:15.921548 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:52:16.921297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:16.921260 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:52:16.921605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:16.921565 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:52:17.845302 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:17.845270 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:17.880453 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:17.880425 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:17.921232 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:17.921207 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:52:17.921362 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:17.921251 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:52:18.397691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:18.397664 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:52:18.921918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:18.921886 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 30 15:52:18.922092 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:18.921937 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 30 15:52:19.925804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:19.925775 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:19.926295 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:19.926152 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:19.927002 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:19.926924 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:52:21.966453 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:52:21.966408 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:52:24.210996 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:24.210909 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:52:27.832256 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.832223 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79"] Oct 30 15:52:27.836516 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.836497 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.857785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.857757 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79"] Oct 30 15:52:27.865019 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.864995 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-dshm\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.865160 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.865136 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tmp-dir\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.865238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.865215 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-model-cache\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.865297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.865268 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82qws\" (UniqueName: \"kubernetes.io/projected/6dba1878-532c-4b2c-ac0f-b9a22c88410b-kube-api-access-82qws\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.865355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.865300 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-home\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.865446 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.865429 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tls-certs\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.867224 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.867204 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c129d1\"/\"kube-root-ca.crt\"" Oct 30 15:52:27.867330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.867240 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c129d1\"/\"default-dockercfg-kc6fq\"" Oct 30 15:52:27.867330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.867310 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c129d1\"/\"premium-w6-c129d1-kserve-self-signed-certs\"" Oct 30 15:52:27.867444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.867362 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-c129d1\"/\"openshift-service-ca.crt\"" Oct 30 15:52:27.966877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.966834 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-dshm\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.966985 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tmp-dir\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.967020 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-model-cache\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.967057 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82qws\" (UniqueName: \"kubernetes.io/projected/6dba1878-532c-4b2c-ac0f-b9a22c88410b-kube-api-access-82qws\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.967085 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-home\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967194 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.967178 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tls-certs\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.967390 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-model-cache\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.967408 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tmp-dir\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.967610 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.967588 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-home\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.969222 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.969199 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-dshm\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.969785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.969764 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tls-certs\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:27.991736 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:27.991707 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82qws\" (UniqueName: \"kubernetes.io/projected/6dba1878-532c-4b2c-ac0f-b9a22c88410b-kube-api-access-82qws\") pod \"premium-w6-c129d1-kserve-5f999d9bf8-fvr79\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:28.151203 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:28.151175 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:28.465337 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:52:28.465308 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dba1878_532c_4b2c_ac0f_b9a22c88410b.slice/crio-98e5accffab820ec7e80481e13caf7aae25caa08d379c0bd871771d7ccc63c16 WatchSource:0}: Error finding container 98e5accffab820ec7e80481e13caf7aae25caa08d379c0bd871771d7ccc63c16: Status 404 returned error can't find the container with id 98e5accffab820ec7e80481e13caf7aae25caa08d379c0bd871771d7ccc63c16 Oct 30 15:52:28.465747 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:28.465723 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79"] Oct 30 15:52:28.560875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:28.560815 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" event={"ID":"6dba1878-532c-4b2c-ac0f-b9a22c88410b","Type":"ContainerStarted","Data":"f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37"} Oct 30 15:52:28.561014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:28.560891 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" event={"ID":"6dba1878-532c-4b2c-ac0f-b9a22c88410b","Type":"ContainerStarted","Data":"98e5accffab820ec7e80481e13caf7aae25caa08d379c0bd871771d7ccc63c16"} Oct 30 15:52:28.605408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:28.605363 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podStartSLOduration=1.605350222 podStartE2EDuration="1.605350222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:28.605134104 +0000 UTC m=+857.304407272" watchObservedRunningTime="2026-10-30 15:52:28.605350222 +0000 UTC m=+857.304623379" Oct 30 15:52:29.151667 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:29.151634 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:37.670126 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.670093 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv"] Oct 30 15:52:37.685450 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.685419 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.691107 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.691073 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv"] Oct 30 15:52:37.701942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.701918 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:52:37.702078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.701985 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:52:37.715858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.715802 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:52:37.715858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.715850 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:52:37.845083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.845048 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.845276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.845126 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxjh\" (UniqueName: \"kubernetes.io/projected/38b9aece-7669-4520-a8e0-5cd3fcea32a5-kube-api-access-snxjh\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.845276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.845267 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.946233 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.946154 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxjh\" (UniqueName: \"kubernetes.io/projected/38b9aece-7669-4520-a8e0-5cd3fcea32a5-kube-api-access-snxjh\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.946403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.946254 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.946403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.946302 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.947031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.947005 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.948662 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.948637 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:37.971002 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:37.970978 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxjh\" (UniqueName: \"kubernetes.io/projected/38b9aece-7669-4520-a8e0-5cd3fcea32a5-kube-api-access-snxjh\") pod \"maas-api-revoke-sub-4d41d55d-72dtv\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:38.003678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:38.003656 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:38.152190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:38.152159 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:38.164132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:38.164110 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:38.307782 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:38.307752 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv"] Oct 30 15:52:38.308213 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:52:38.308193 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b9aece_7669_4520_a8e0_5cd3fcea32a5.slice/crio-39898371b0a90f6c1a014dc96586117374481ab0f7e0e0fe9db6e70d4300364a WatchSource:0}: Error finding container 39898371b0a90f6c1a014dc96586117374481ab0f7e0e0fe9db6e70d4300364a: Status 404 returned error can't find the container with id 39898371b0a90f6c1a014dc96586117374481ab0f7e0e0fe9db6e70d4300364a Oct 30 15:52:38.624581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:38.624548 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" event={"ID":"38b9aece-7669-4520-a8e0-5cd3fcea32a5","Type":"ContainerStarted","Data":"39898371b0a90f6c1a014dc96586117374481ab0f7e0e0fe9db6e70d4300364a"} Oct 30 15:52:38.635979 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:38.635958 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:52:41.740388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:41.740292 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" event={"ID":"38b9aece-7669-4520-a8e0-5cd3fcea32a5","Type":"ContainerStarted","Data":"c591e1955a4c757786653eb9334abc4b101c79a552ce00a4dd7c2aa3fcaba67e"} Oct 30 15:52:42.648850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:42.640905 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:52:42.680035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:42.679904 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:52:42.783512 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:42.782950 4401 generic.go:356] "Generic (PLEG): container finished" podID="38b9aece-7669-4520-a8e0-5cd3fcea32a5" containerID="c591e1955a4c757786653eb9334abc4b101c79a552ce00a4dd7c2aa3fcaba67e" exitCode=0 Oct 30 15:52:42.783512 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:42.783109 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" event={"ID":"38b9aece-7669-4520-a8e0-5cd3fcea32a5","Type":"ContainerDied","Data":"c591e1955a4c757786653eb9334abc4b101c79a552ce00a4dd7c2aa3fcaba67e"} Oct 30 15:52:42.989903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:42.988933 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:52:43.263010 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:43.262907 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:52:43.393847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:43.393444 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:52:43.930690 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:43.930665 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:44.063770 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.063675 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-openshift-ca\") pod \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " Oct 30 15:52:44.063985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.063768 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-xks-ca\") pod \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " Oct 30 15:52:44.063985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.063887 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38b9aece-7669-4520-a8e0-5cd3fcea32a5-kube-api-access-snxjh\" (UniqueName: \"kubernetes.io/projected/38b9aece-7669-4520-a8e0-5cd3fcea32a5-kube-api-access-snxjh\") pod \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\" (UID: \"38b9aece-7669-4520-a8e0-5cd3fcea32a5\") " Oct 30 15:52:44.064113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.064086 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-openshift-ca" pod "38b9aece-7669-4520-a8e0-5cd3fcea32a5" (UID: "38b9aece-7669-4520-a8e0-5cd3fcea32a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:52:44.064384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.064364 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:44.065968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.065947 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-xks-ca" pod "38b9aece-7669-4520-a8e0-5cd3fcea32a5" (UID: "38b9aece-7669-4520-a8e0-5cd3fcea32a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:52:44.066092 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.066073 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38b9aece-7669-4520-a8e0-5cd3fcea32a5-kube-api-access-snxjh" pod "38b9aece-7669-4520-a8e0-5cd3fcea32a5" (UID: "38b9aece-7669-4520-a8e0-5cd3fcea32a5"). InnerVolumeSpecName "kube-api-access-snxjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:52:44.165686 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.165649 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38b9aece-7669-4520-a8e0-5cd3fcea32a5-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:44.165686 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.165678 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snxjh\" (UniqueName: \"kubernetes.io/projected/38b9aece-7669-4520-a8e0-5cd3fcea32a5-kube-api-access-snxjh\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:44.798182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.798153 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" Oct 30 15:52:44.798355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.798186 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv" event={"ID":"38b9aece-7669-4520-a8e0-5cd3fcea32a5","Type":"ContainerDied","Data":"39898371b0a90f6c1a014dc96586117374481ab0f7e0e0fe9db6e70d4300364a"} Oct 30 15:52:44.798355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:44.798231 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39898371b0a90f6c1a014dc96586117374481ab0f7e0e0fe9db6e70d4300364a" Oct 30 15:52:50.616953 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.616918 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54"] Oct 30 15:52:50.617515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.617501 4401 cpu_manager.go:414] "containerMap: removing container" podUID="38b9aece-7669-4520-a8e0-5cd3fcea32a5" containerName="revoke-keys" Oct 30 15:52:50.617579 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.617516 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38b9aece-7669-4520-a8e0-5cd3fcea32a5" containerName="revoke-keys" Oct 30 15:52:50.617634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.617624 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="38b9aece-7669-4520-a8e0-5cd3fcea32a5" containerName="revoke-keys" Oct 30 15:52:50.621241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.621222 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.639570 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.639532 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54"] Oct 30 15:52:50.648841 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.648804 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:52:50.648972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.648861 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:52:50.649049 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.649027 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:52:50.649161 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.649097 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:52:50.753667 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.753636 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.753845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.753696 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.753845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.753816 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrqf\" (UniqueName: \"kubernetes.io/projected/6edaaaab-ad02-4621-b9ad-83a2196fc572-kube-api-access-zvrqf\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.855455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.855421 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.855639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.855480 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.855639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.855514 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrqf\" (UniqueName: \"kubernetes.io/projected/6edaaaab-ad02-4621-b9ad-83a2196fc572-kube-api-access-zvrqf\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.856222 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.856196 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.857663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.857637 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.881603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.881543 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrqf\" (UniqueName: \"kubernetes.io/projected/6edaaaab-ad02-4621-b9ad-83a2196fc572-kube-api-access-zvrqf\") pod \"maas-api-revoke-sub-60954d45-cbm54\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:50.936359 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:50.936330 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:51.241056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:51.241022 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54"] Oct 30 15:52:51.241534 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:52:51.241507 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edaaaab_ad02_4621_b9ad_83a2196fc572.slice/crio-7b05083be36b7dfa78b854311470ed04d5fa831bb6aaa09bff63dc3a65f29e01 WatchSource:0}: Error finding container 7b05083be36b7dfa78b854311470ed04d5fa831bb6aaa09bff63dc3a65f29e01: Status 404 returned error can't find the container with id 7b05083be36b7dfa78b854311470ed04d5fa831bb6aaa09bff63dc3a65f29e01 Oct 30 15:52:51.878125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:51.878088 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" event={"ID":"6edaaaab-ad02-4621-b9ad-83a2196fc572","Type":"ContainerStarted","Data":"1dcdc1c7eb301d3e59c00bd809a9be9a5e9e4c73a07d94860348ab9a6762986a"} Oct 30 15:52:51.878579 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:51.878145 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" event={"ID":"6edaaaab-ad02-4621-b9ad-83a2196fc572","Type":"ContainerStarted","Data":"7b05083be36b7dfa78b854311470ed04d5fa831bb6aaa09bff63dc3a65f29e01"} Oct 30 15:52:51.933739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:51.933688 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" podStartSLOduration=1.9336753770000001 podStartE2EDuration="1.933675377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:51.930752605 +0000 UTC m=+880.630025762" watchObservedRunningTime="2026-10-30 15:52:51.933675377 +0000 UTC m=+880.632948535" Oct 30 15:52:51.967318 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:52:51.967285 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:52:52.223409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.223375 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns"] Oct 30 15:52:52.227685 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.227665 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.246659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.246632 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns"] Oct 30 15:52:52.259057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.259026 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb"] Oct 30 15:52:52.264176 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.264154 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.280861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.280086 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb"] Oct 30 15:52:52.392463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.392429 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.392663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.392470 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmw5\" (UniqueName: \"kubernetes.io/projected/98153aa6-6b69-4e94-b9ff-e8527e72f819-kube-api-access-bzmw5\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.392663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.392497 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.392663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.392545 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.392663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.392645 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbtb6\" (UniqueName: \"kubernetes.io/projected/258a70c7-fa54-4eef-82fb-6b893604b585-kube-api-access-jbtb6\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.392912 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.392773 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.493908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.493813 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.494070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.493975 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.494070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.494015 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmw5\" (UniqueName: \"kubernetes.io/projected/98153aa6-6b69-4e94-b9ff-e8527e72f819-kube-api-access-bzmw5\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.494070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.494040 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.494070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.494061 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.494277 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.494094 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbtb6\" (UniqueName: \"kubernetes.io/projected/258a70c7-fa54-4eef-82fb-6b893604b585-kube-api-access-jbtb6\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.494707 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.494687 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.494816 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.494793 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.496267 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.496245 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.496373 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.496281 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.519183 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.519161 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmw5\" (UniqueName: \"kubernetes.io/projected/98153aa6-6b69-4e94-b9ff-e8527e72f819-kube-api-access-bzmw5\") pod \"maas-api-revoke-sub-230431fe-rzjns\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.519281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.519225 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbtb6\" (UniqueName: \"kubernetes.io/projected/258a70c7-fa54-4eef-82fb-6b893604b585-kube-api-access-jbtb6\") pod \"maas-api-revoke-sub-f692dfd1-6g9fb\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.560404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.560381 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:52.589559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.589536 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:52.866274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.866245 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns"] Oct 30 15:52:52.866432 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:52:52.866406 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98153aa6_6b69_4e94_b9ff_e8527e72f819.slice/crio-058c2eac4361e108d0b3c3fd6a3d4923fd24717be5aaf60e17cce7b1d1abf549 WatchSource:0}: Error finding container 058c2eac4361e108d0b3c3fd6a3d4923fd24717be5aaf60e17cce7b1d1abf549: Status 404 returned error can't find the container with id 058c2eac4361e108d0b3c3fd6a3d4923fd24717be5aaf60e17cce7b1d1abf549 Oct 30 15:52:52.886656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.886626 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" event={"ID":"98153aa6-6b69-4e94-b9ff-e8527e72f819","Type":"ContainerStarted","Data":"058c2eac4361e108d0b3c3fd6a3d4923fd24717be5aaf60e17cce7b1d1abf549"} Oct 30 15:52:52.888413 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.888385 4401 generic.go:356] "Generic (PLEG): container finished" podID="6edaaaab-ad02-4621-b9ad-83a2196fc572" containerID="1dcdc1c7eb301d3e59c00bd809a9be9a5e9e4c73a07d94860348ab9a6762986a" exitCode=0 Oct 30 15:52:52.888535 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.888496 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" event={"ID":"6edaaaab-ad02-4621-b9ad-83a2196fc572","Type":"ContainerDied","Data":"1dcdc1c7eb301d3e59c00bd809a9be9a5e9e4c73a07d94860348ab9a6762986a"} Oct 30 15:52:52.919433 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:52:52.919407 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258a70c7_fa54_4eef_82fb_6b893604b585.slice/crio-bea45691d9e8d311cb042a1bd5bc0ff5a299cbc34df4106c61c9a05b982cdaef WatchSource:0}: Error finding container bea45691d9e8d311cb042a1bd5bc0ff5a299cbc34df4106c61c9a05b982cdaef: Status 404 returned error can't find the container with id bea45691d9e8d311cb042a1bd5bc0ff5a299cbc34df4106c61c9a05b982cdaef Oct 30 15:52:52.919560 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:52.919526 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb"] Oct 30 15:52:53.896773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:53.896739 4401 generic.go:356] "Generic (PLEG): container finished" podID="98153aa6-6b69-4e94-b9ff-e8527e72f819" containerID="16d2e0026ca55577a41e90e5e0b961eca2ac92d01b6f9a74dffaf618aba12425" exitCode=0 Oct 30 15:52:53.897144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:53.896819 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" event={"ID":"98153aa6-6b69-4e94-b9ff-e8527e72f819","Type":"ContainerDied","Data":"16d2e0026ca55577a41e90e5e0b961eca2ac92d01b6f9a74dffaf618aba12425"} Oct 30 15:52:53.898427 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:53.898406 4401 generic.go:356] "Generic (PLEG): container finished" podID="258a70c7-fa54-4eef-82fb-6b893604b585" containerID="d18d86d4ad4b6e71156fe0eff33976b3f1d70d27d511453bc713b12dd2c0bb8c" exitCode=0 Oct 30 15:52:53.898541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:53.898486 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" event={"ID":"258a70c7-fa54-4eef-82fb-6b893604b585","Type":"ContainerDied","Data":"d18d86d4ad4b6e71156fe0eff33976b3f1d70d27d511453bc713b12dd2c0bb8c"} Oct 30 15:52:53.898541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:53.898529 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" event={"ID":"258a70c7-fa54-4eef-82fb-6b893604b585","Type":"ContainerStarted","Data":"bea45691d9e8d311cb042a1bd5bc0ff5a299cbc34df4106c61c9a05b982cdaef"} Oct 30 15:52:54.053517 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.053495 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:54.213746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.213675 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-openshift-ca\") pod \"6edaaaab-ad02-4621-b9ad-83a2196fc572\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " Oct 30 15:52:54.213746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.213739 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-xks-ca\") pod \"6edaaaab-ad02-4621-b9ad-83a2196fc572\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " Oct 30 15:52:54.213972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.213765 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edaaaab-ad02-4621-b9ad-83a2196fc572-kube-api-access-zvrqf\" (UniqueName: \"kubernetes.io/projected/6edaaaab-ad02-4621-b9ad-83a2196fc572-kube-api-access-zvrqf\") pod \"6edaaaab-ad02-4621-b9ad-83a2196fc572\" (UID: \"6edaaaab-ad02-4621-b9ad-83a2196fc572\") " Oct 30 15:52:54.214036 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.214002 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-openshift-ca" pod "6edaaaab-ad02-4621-b9ad-83a2196fc572" (UID: "6edaaaab-ad02-4621-b9ad-83a2196fc572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:52:54.214199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.214181 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:54.215792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.215773 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-xks-ca" pod "6edaaaab-ad02-4621-b9ad-83a2196fc572" (UID: "6edaaaab-ad02-4621-b9ad-83a2196fc572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:52:54.215855 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.215780 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edaaaab-ad02-4621-b9ad-83a2196fc572-kube-api-access-zvrqf" pod "6edaaaab-ad02-4621-b9ad-83a2196fc572" (UID: "6edaaaab-ad02-4621-b9ad-83a2196fc572"). InnerVolumeSpecName "kube-api-access-zvrqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:52:54.315348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.315317 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6edaaaab-ad02-4621-b9ad-83a2196fc572-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:54.315348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.315341 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvrqf\" (UniqueName: \"kubernetes.io/projected/6edaaaab-ad02-4621-b9ad-83a2196fc572-kube-api-access-zvrqf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:54.905977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.905949 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" Oct 30 15:52:54.906377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.905987 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54" event={"ID":"6edaaaab-ad02-4621-b9ad-83a2196fc572","Type":"ContainerDied","Data":"7b05083be36b7dfa78b854311470ed04d5fa831bb6aaa09bff63dc3a65f29e01"} Oct 30 15:52:54.906377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:54.906033 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b05083be36b7dfa78b854311470ed04d5fa831bb6aaa09bff63dc3a65f29e01" Oct 30 15:52:55.128306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.128275 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:55.152005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.151980 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:55.234428 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234352 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98153aa6-6b69-4e94-b9ff-e8527e72f819-kube-api-access-bzmw5\" (UniqueName: \"kubernetes.io/projected/98153aa6-6b69-4e94-b9ff-e8527e72f819-kube-api-access-bzmw5\") pod \"98153aa6-6b69-4e94-b9ff-e8527e72f819\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " Oct 30 15:52:55.234428 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234413 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-openshift-ca\") pod \"258a70c7-fa54-4eef-82fb-6b893604b585\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " Oct 30 15:52:55.234658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234448 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-xks-ca\") pod \"98153aa6-6b69-4e94-b9ff-e8527e72f819\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " Oct 30 15:52:55.234658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234541 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-openshift-ca\") pod \"98153aa6-6b69-4e94-b9ff-e8527e72f819\" (UID: \"98153aa6-6b69-4e94-b9ff-e8527e72f819\") " Oct 30 15:52:55.234658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234636 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/258a70c7-fa54-4eef-82fb-6b893604b585-kube-api-access-jbtb6\" (UniqueName: \"kubernetes.io/projected/258a70c7-fa54-4eef-82fb-6b893604b585-kube-api-access-jbtb6\") pod \"258a70c7-fa54-4eef-82fb-6b893604b585\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " Oct 30 15:52:55.234867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234759 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-xks-ca\") pod \"258a70c7-fa54-4eef-82fb-6b893604b585\" (UID: \"258a70c7-fa54-4eef-82fb-6b893604b585\") " Oct 30 15:52:55.234867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234775 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-openshift-ca" pod "258a70c7-fa54-4eef-82fb-6b893604b585" (UID: "258a70c7-fa54-4eef-82fb-6b893604b585"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:52:55.235010 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.234987 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-openshift-ca" pod "98153aa6-6b69-4e94-b9ff-e8527e72f819" (UID: "98153aa6-6b69-4e94-b9ff-e8527e72f819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:52:55.235372 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.235331 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:55.235372 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.235360 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:55.237185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.237158 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-xks-ca" pod "98153aa6-6b69-4e94-b9ff-e8527e72f819" (UID: "98153aa6-6b69-4e94-b9ff-e8527e72f819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:52:55.237269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.237195 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-xks-ca" pod "258a70c7-fa54-4eef-82fb-6b893604b585" (UID: "258a70c7-fa54-4eef-82fb-6b893604b585"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:52:55.237306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.237273 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98153aa6-6b69-4e94-b9ff-e8527e72f819-kube-api-access-bzmw5" pod "98153aa6-6b69-4e94-b9ff-e8527e72f819" (UID: "98153aa6-6b69-4e94-b9ff-e8527e72f819"). InnerVolumeSpecName "kube-api-access-bzmw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:52:55.237361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.237342 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258a70c7-fa54-4eef-82fb-6b893604b585-kube-api-access-jbtb6" pod "258a70c7-fa54-4eef-82fb-6b893604b585" (UID: "258a70c7-fa54-4eef-82fb-6b893604b585"). InnerVolumeSpecName "kube-api-access-jbtb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:52:55.336712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.336689 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbtb6\" (UniqueName: \"kubernetes.io/projected/258a70c7-fa54-4eef-82fb-6b893604b585-kube-api-access-jbtb6\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:55.336712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.336710 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/258a70c7-fa54-4eef-82fb-6b893604b585-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:55.336893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.336719 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzmw5\" (UniqueName: \"kubernetes.io/projected/98153aa6-6b69-4e94-b9ff-e8527e72f819-kube-api-access-bzmw5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:55.336893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.336728 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98153aa6-6b69-4e94-b9ff-e8527e72f819-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:52:55.913815 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.913787 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" Oct 30 15:52:55.914249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.913781 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns" event={"ID":"98153aa6-6b69-4e94-b9ff-e8527e72f819","Type":"ContainerDied","Data":"058c2eac4361e108d0b3c3fd6a3d4923fd24717be5aaf60e17cce7b1d1abf549"} Oct 30 15:52:55.914249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.913967 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058c2eac4361e108d0b3c3fd6a3d4923fd24717be5aaf60e17cce7b1d1abf549" Oct 30 15:52:55.915241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.915224 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" Oct 30 15:52:55.915340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.915222 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb" event={"ID":"258a70c7-fa54-4eef-82fb-6b893604b585","Type":"ContainerDied","Data":"bea45691d9e8d311cb042a1bd5bc0ff5a299cbc34df4106c61c9a05b982cdaef"} Oct 30 15:52:55.915379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:55.915340 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bea45691d9e8d311cb042a1bd5bc0ff5a299cbc34df4106c61c9a05b982cdaef" Oct 30 15:52:56.300951 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:52:56.300878 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:53:04.917679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.917643 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r"] Oct 30 15:53:04.918271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918256 4401 cpu_manager.go:414] "containerMap: removing container" podUID="98153aa6-6b69-4e94-b9ff-e8527e72f819" containerName="revoke-keys" Oct 30 15:53:04.918312 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918271 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98153aa6-6b69-4e94-b9ff-e8527e72f819" containerName="revoke-keys" Oct 30 15:53:04.918312 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918294 4401 cpu_manager.go:414] "containerMap: removing container" podUID="6edaaaab-ad02-4621-b9ad-83a2196fc572" containerName="revoke-keys" Oct 30 15:53:04.918312 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918299 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edaaaab-ad02-4621-b9ad-83a2196fc572" containerName="revoke-keys" Oct 30 15:53:04.918414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918314 4401 cpu_manager.go:414] "containerMap: removing container" podUID="258a70c7-fa54-4eef-82fb-6b893604b585" containerName="revoke-keys" Oct 30 15:53:04.918414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918329 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="258a70c7-fa54-4eef-82fb-6b893604b585" containerName="revoke-keys" Oct 30 15:53:04.918470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918442 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="98153aa6-6b69-4e94-b9ff-e8527e72f819" containerName="revoke-keys" Oct 30 15:53:04.918470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918456 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edaaaab-ad02-4621-b9ad-83a2196fc572" containerName="revoke-keys" Oct 30 15:53:04.918470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.918463 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="258a70c7-fa54-4eef-82fb-6b893604b585" containerName="revoke-keys" Oct 30 15:53:04.921978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.921961 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:04.929526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.929500 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4qr2\" (UniqueName: \"kubernetes.io/projected/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-kube-api-access-l4qr2\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:04.929676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.929655 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:04.929740 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.929729 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:04.943495 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.943465 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r"] Oct 30 15:53:04.967867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.951712 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:53:04.967867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.952494 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:53:04.967867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.952553 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:53:04.967867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:04.961024 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:53:05.031356 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.031327 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:05.031575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.031383 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:05.031575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.031509 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4qr2\" (UniqueName: \"kubernetes.io/projected/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-kube-api-access-l4qr2\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:05.032062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.032041 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:05.033613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.033592 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:05.060685 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.060658 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4qr2\" (UniqueName: \"kubernetes.io/projected/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-kube-api-access-l4qr2\") pod \"maas-api-revoke-sub-a22a711e-w985r\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:05.239550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.239470 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:05.560330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.560297 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r"] Oct 30 15:53:05.560649 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:05.560626 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ec4fa9_c902_412a_9b67_4db5cdcc5f92.slice/crio-000bd5ae8315f161b76be9b850463b404055b0dbafae2ad840050d708fa2c28a WatchSource:0}: Error finding container 000bd5ae8315f161b76be9b850463b404055b0dbafae2ad840050d708fa2c28a: Status 404 returned error can't find the container with id 000bd5ae8315f161b76be9b850463b404055b0dbafae2ad840050d708fa2c28a Oct 30 15:53:05.833108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.833021 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6"] Oct 30 15:53:05.838451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.838432 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:05.863173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.856355 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6"] Oct 30 15:53:05.941442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.941404 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:05.941860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.941537 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwczw\" (UniqueName: \"kubernetes.io/projected/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-kube-api-access-vwczw\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:05.941860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:05.941588 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.016237 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.016206 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" event={"ID":"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92","Type":"ContainerStarted","Data":"05ebbe6fc3935be6e774ac9996d7f6ca7bc8b7b128205c398e6a4427f6e0e147"} Oct 30 15:53:06.016410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.016268 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" event={"ID":"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92","Type":"ContainerStarted","Data":"000bd5ae8315f161b76be9b850463b404055b0dbafae2ad840050d708fa2c28a"} Oct 30 15:53:06.042569 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.042533 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.042726 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.042707 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.042876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.042851 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwczw\" (UniqueName: \"kubernetes.io/projected/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-kube-api-access-vwczw\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.043194 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.043173 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.045187 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.045166 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.068026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.068005 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwczw\" (UniqueName: \"kubernetes.io/projected/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-kube-api-access-vwczw\") pod \"maas-api-revoke-sub-e5059029-r6qg6\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.071098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.071060 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" podStartSLOduration=2.071048327 podStartE2EDuration="2.071048327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:06.069371622 +0000 UTC m=+894.768644774" watchObservedRunningTime="2026-10-30 15:53:06.071048327 +0000 UTC m=+894.770321486" Oct 30 15:53:06.153714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.153689 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:06.459919 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:06.459886 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9fbe44_0c1b_4ca4_afe8_1aea86842094.slice/crio-8625b93dafb1f889f41d460e437312e935b572ed717554e2e8a08ec8bd7b6004 WatchSource:0}: Error finding container 8625b93dafb1f889f41d460e437312e935b572ed717554e2e8a08ec8bd7b6004: Status 404 returned error can't find the container with id 8625b93dafb1f889f41d460e437312e935b572ed717554e2e8a08ec8bd7b6004 Oct 30 15:53:06.460141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:06.460119 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6"] Oct 30 15:53:07.024770 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.024690 4401 generic.go:356] "Generic (PLEG): container finished" podID="b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" containerID="05ebbe6fc3935be6e774ac9996d7f6ca7bc8b7b128205c398e6a4427f6e0e147" exitCode=0 Oct 30 15:53:07.025203 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.024775 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" event={"ID":"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92","Type":"ContainerDied","Data":"05ebbe6fc3935be6e774ac9996d7f6ca7bc8b7b128205c398e6a4427f6e0e147"} Oct 30 15:53:07.026410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.026387 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" event={"ID":"aa9fbe44-0c1b-4ca4-afe8-1aea86842094","Type":"ContainerStarted","Data":"ffe6fac3124b55a6c445a298f707145fbf19bd80ede82a98e20a10e605f017aa"} Oct 30 15:53:07.026524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.026431 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" event={"ID":"aa9fbe44-0c1b-4ca4-afe8-1aea86842094","Type":"ContainerStarted","Data":"8625b93dafb1f889f41d460e437312e935b572ed717554e2e8a08ec8bd7b6004"} Oct 30 15:53:07.116893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.116841 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" podStartSLOduration=2.116818175 podStartE2EDuration="2.116818175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:07.116212264 +0000 UTC m=+895.815485422" watchObservedRunningTime="2026-10-30 15:53:07.116818175 +0000 UTC m=+895.816091332" Oct 30 15:53:07.691867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.691780 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk"] Oct 30 15:53:07.700726 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.700694 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.714386 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.714353 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk"] Oct 30 15:53:07.761967 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.761937 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.762101 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.762003 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.762169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.762102 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxbfh\" (UniqueName: \"kubernetes.io/projected/eae5fda4-df24-4926-9ed7-de6b988f6ace-kube-api-access-sxbfh\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.863306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.863266 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.863480 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.863357 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.863480 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.863412 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxbfh\" (UniqueName: \"kubernetes.io/projected/eae5fda4-df24-4926-9ed7-de6b988f6ace-kube-api-access-sxbfh\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.864045 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.864022 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.865572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.865550 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:07.888328 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:07.888306 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxbfh\" (UniqueName: \"kubernetes.io/projected/eae5fda4-df24-4926-9ed7-de6b988f6ace-kube-api-access-sxbfh\") pod \"maas-api-revoke-sub-c65c7b61-zmvrk\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:08.016874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.016760 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:08.037958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.037925 4401 generic.go:356] "Generic (PLEG): container finished" podID="aa9fbe44-0c1b-4ca4-afe8-1aea86842094" containerID="ffe6fac3124b55a6c445a298f707145fbf19bd80ede82a98e20a10e605f017aa" exitCode=0 Oct 30 15:53:08.038297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.038006 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" event={"ID":"aa9fbe44-0c1b-4ca4-afe8-1aea86842094","Type":"ContainerDied","Data":"ffe6fac3124b55a6c445a298f707145fbf19bd80ede82a98e20a10e605f017aa"} Oct 30 15:53:08.451275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.451242 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk"] Oct 30 15:53:08.522170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.522140 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:08.718895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.711759 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-openshift-ca\") pod \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " Oct 30 15:53:08.718895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.713962 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-openshift-ca" pod "b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" (UID: "b4ec4fa9-c902-412a-9b67-4db5cdcc5f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:08.718895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.716115 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-kube-api-access-l4qr2\" (UniqueName: \"kubernetes.io/projected/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-kube-api-access-l4qr2\") pod \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " Oct 30 15:53:08.718895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.716308 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-xks-ca\") pod \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\" (UID: \"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92\") " Oct 30 15:53:08.718895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.716808 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:08.724970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.724924 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-kube-api-access-l4qr2" pod "b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" (UID: "b4ec4fa9-c902-412a-9b67-4db5cdcc5f92"). InnerVolumeSpecName "kube-api-access-l4qr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:08.726114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.726078 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-xks-ca" pod "b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" (UID: "b4ec4fa9-c902-412a-9b67-4db5cdcc5f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:08.825848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.824902 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4qr2\" (UniqueName: \"kubernetes.io/projected/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-kube-api-access-l4qr2\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:08.825848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.824944 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:08.926572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.926524 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj"] Oct 30 15:53:08.937602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.937570 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" containerName="revoke-keys" Oct 30 15:53:08.937849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.937818 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" containerName="revoke-keys" Oct 30 15:53:08.938206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.938192 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" containerName="revoke-keys" Oct 30 15:53:08.944942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.944918 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj"] Oct 30 15:53:08.945157 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:08.945142 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.068870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.046654 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.068870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.046737 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.068870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.046814 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8jsb\" (UniqueName: \"kubernetes.io/projected/d5e774eb-c937-4454-842d-f83beaca68f2-kube-api-access-n8jsb\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.068870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.056790 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" event={"ID":"eae5fda4-df24-4926-9ed7-de6b988f6ace","Type":"ContainerStarted","Data":"2ffcd6df6f44a4b94d52eba685a526af132a0136431d7a193e0e2c93db47dc31"} Oct 30 15:53:09.073599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.073561 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" Oct 30 15:53:09.075807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.075769 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r" event={"ID":"b4ec4fa9-c902-412a-9b67-4db5cdcc5f92","Type":"ContainerDied","Data":"000bd5ae8315f161b76be9b850463b404055b0dbafae2ad840050d708fa2c28a"} Oct 30 15:53:09.076039 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.076017 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="000bd5ae8315f161b76be9b850463b404055b0dbafae2ad840050d708fa2c28a" Oct 30 15:53:09.152379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.152345 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.169849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.161302 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.169849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.161513 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.169849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.162451 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8jsb\" (UniqueName: \"kubernetes.io/projected/d5e774eb-c937-4454-842d-f83beaca68f2-kube-api-access-n8jsb\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.242903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.202676 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8jsb\" (UniqueName: \"kubernetes.io/projected/d5e774eb-c937-4454-842d-f83beaca68f2-kube-api-access-n8jsb\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.242903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.203166 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e3682f-8flbj\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:09.399892 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:09.395578 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:10.143916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.143491 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" event={"ID":"aa9fbe44-0c1b-4ca4-afe8-1aea86842094","Type":"ContainerDied","Data":"8625b93dafb1f889f41d460e437312e935b572ed717554e2e8a08ec8bd7b6004"} Oct 30 15:53:10.143916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.143556 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8625b93dafb1f889f41d460e437312e935b572ed717554e2e8a08ec8bd7b6004" Oct 30 15:53:10.147860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.147739 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" event={"ID":"eae5fda4-df24-4926-9ed7-de6b988f6ace","Type":"ContainerStarted","Data":"f7711b3d6ec882568db0b77a6e1fd69869c9d93a70972d1a90c90652dfe9d83a"} Oct 30 15:53:10.198917 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.197013 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:10.198917 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.197108 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" podStartSLOduration=3.197090515 podStartE2EDuration="3.197090515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:10.196081125 +0000 UTC m=+898.895354285" watchObservedRunningTime="2026-10-30 15:53:10.197090515 +0000 UTC m=+898.896363672" Oct 30 15:53:10.298931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.298426 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-openshift-ca\") pod \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " Oct 30 15:53:10.298931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.298562 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-kube-api-access-vwczw\" (UniqueName: \"kubernetes.io/projected/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-kube-api-access-vwczw\") pod \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " Oct 30 15:53:10.298931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.298678 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-xks-ca\") pod \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\" (UID: \"aa9fbe44-0c1b-4ca4-afe8-1aea86842094\") " Oct 30 15:53:10.300682 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.300376 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-openshift-ca" pod "aa9fbe44-0c1b-4ca4-afe8-1aea86842094" (UID: "aa9fbe44-0c1b-4ca4-afe8-1aea86842094"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:10.325175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.324956 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-kube-api-access-vwczw" pod "aa9fbe44-0c1b-4ca4-afe8-1aea86842094" (UID: "aa9fbe44-0c1b-4ca4-afe8-1aea86842094"). InnerVolumeSpecName "kube-api-access-vwczw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:10.343333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.343295 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-xks-ca" pod "aa9fbe44-0c1b-4ca4-afe8-1aea86842094" (UID: "aa9fbe44-0c1b-4ca4-afe8-1aea86842094"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:10.417721 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.417629 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:10.417721 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.417661 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwczw\" (UniqueName: \"kubernetes.io/projected/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-kube-api-access-vwczw\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:10.417721 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.417671 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9fbe44-0c1b-4ca4-afe8-1aea86842094-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:10.452377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:10.452348 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj"] Oct 30 15:53:10.452476 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:10.452370 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e774eb_c937_4454_842d_f83beaca68f2.slice/crio-2e4dcd8f0f3d22ab222c62abdb710d6ddfd730ce672bf2a441cd674aeb0833ae WatchSource:0}: Error finding container 2e4dcd8f0f3d22ab222c62abdb710d6ddfd730ce672bf2a441cd674aeb0833ae: Status 404 returned error can't find the container with id 2e4dcd8f0f3d22ab222c62abdb710d6ddfd730ce672bf2a441cd674aeb0833ae Oct 30 15:53:11.156374 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.156328 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" event={"ID":"d5e774eb-c937-4454-842d-f83beaca68f2","Type":"ContainerStarted","Data":"92ef94173977acda1e98d200a3f2ad82ab98916cd2bceae0983e7d76d6efb074"} Oct 30 15:53:11.156819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.156401 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" event={"ID":"d5e774eb-c937-4454-842d-f83beaca68f2","Type":"ContainerStarted","Data":"2e4dcd8f0f3d22ab222c62abdb710d6ddfd730ce672bf2a441cd674aeb0833ae"} Oct 30 15:53:11.158028 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.158002 4401 generic.go:356] "Generic (PLEG): container finished" podID="eae5fda4-df24-4926-9ed7-de6b988f6ace" containerID="f7711b3d6ec882568db0b77a6e1fd69869c9d93a70972d1a90c90652dfe9d83a" exitCode=0 Oct 30 15:53:11.158128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.158089 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" event={"ID":"eae5fda4-df24-4926-9ed7-de6b988f6ace","Type":"ContainerDied","Data":"f7711b3d6ec882568db0b77a6e1fd69869c9d93a70972d1a90c90652dfe9d83a"} Oct 30 15:53:11.158228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.158214 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6" Oct 30 15:53:11.202412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.202358 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" podStartSLOduration=3.20234483 podStartE2EDuration="3.20234483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:11.200085839 +0000 UTC m=+899.899358996" watchObservedRunningTime="2026-10-30 15:53:11.20234483 +0000 UTC m=+899.901617989" Oct 30 15:53:11.945916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.945879 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj"] Oct 30 15:53:11.946505 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.946491 4401 cpu_manager.go:414] "containerMap: removing container" podUID="aa9fbe44-0c1b-4ca4-afe8-1aea86842094" containerName="revoke-keys" Oct 30 15:53:11.946556 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.946506 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9fbe44-0c1b-4ca4-afe8-1aea86842094" containerName="revoke-keys" Oct 30 15:53:11.946627 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.946613 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9fbe44-0c1b-4ca4-afe8-1aea86842094" containerName="revoke-keys" Oct 30 15:53:11.950268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.950251 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:11.970444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.970385 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-b24e7b-openshift-default-dockercfg-fsj9p\"" Oct 30 15:53:11.988380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:11.988352 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj"] Oct 30 15:53:12.036439 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036408 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036466 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036542 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036575 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xq2\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-kube-api-access-g8xq2\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036618 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036652 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036718 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036764 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.036905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.036892 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138080 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138191 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138235 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138264 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138291 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xq2\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-kube-api-access-g8xq2\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138347 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138372 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138409 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138473 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138700 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138767 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138734 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138813 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138801 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138894 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138874 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.138937 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.138895 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.140534 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.140507 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.140919 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.140900 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.163310 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.163282 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.164088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.164062 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xq2\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-kube-api-access-g8xq2\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.172112 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.172078 4401 generic.go:356] "Generic (PLEG): container finished" podID="d5e774eb-c937-4454-842d-f83beaca68f2" containerID="92ef94173977acda1e98d200a3f2ad82ab98916cd2bceae0983e7d76d6efb074" exitCode=0 Oct 30 15:53:12.172620 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.172589 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" event={"ID":"d5e774eb-c937-4454-842d-f83beaca68f2","Type":"ContainerDied","Data":"92ef94173977acda1e98d200a3f2ad82ab98916cd2bceae0983e7d76d6efb074"} Oct 30 15:53:12.263594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.263563 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:12.307425 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.307400 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:12.340138 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.339809 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae5fda4-df24-4926-9ed7-de6b988f6ace-kube-api-access-sxbfh\" (UniqueName: \"kubernetes.io/projected/eae5fda4-df24-4926-9ed7-de6b988f6ace-kube-api-access-sxbfh\") pod \"eae5fda4-df24-4926-9ed7-de6b988f6ace\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " Oct 30 15:53:12.340138 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.339980 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-openshift-ca\") pod \"eae5fda4-df24-4926-9ed7-de6b988f6ace\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " Oct 30 15:53:12.340324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.340214 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-xks-ca\") pod \"eae5fda4-df24-4926-9ed7-de6b988f6ace\" (UID: \"eae5fda4-df24-4926-9ed7-de6b988f6ace\") " Oct 30 15:53:12.340576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.340540 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-openshift-ca" pod "eae5fda4-df24-4926-9ed7-de6b988f6ace" (UID: "eae5fda4-df24-4926-9ed7-de6b988f6ace"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:12.340881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.340691 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:12.342986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.342432 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-xks-ca" pod "eae5fda4-df24-4926-9ed7-de6b988f6ace" (UID: "eae5fda4-df24-4926-9ed7-de6b988f6ace"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:12.343570 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.343540 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae5fda4-df24-4926-9ed7-de6b988f6ace-kube-api-access-sxbfh" pod "eae5fda4-df24-4926-9ed7-de6b988f6ace" (UID: "eae5fda4-df24-4926-9ed7-de6b988f6ace"). InnerVolumeSpecName "kube-api-access-sxbfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:12.441832 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.441789 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxbfh\" (UniqueName: \"kubernetes.io/projected/eae5fda4-df24-4926-9ed7-de6b988f6ace-kube-api-access-sxbfh\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:12.441832 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.441818 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eae5fda4-df24-4926-9ed7-de6b988f6ace-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:12.600952 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:12.600913 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b4e65e_4696_4b8d_a5a9_f6ab671149e8.slice/crio-99cd352f9f690c35ab6bcbbeac911d6699a2dc72a867c3b25b185d7cd426a438 WatchSource:0}: Error finding container 99cd352f9f690c35ab6bcbbeac911d6699a2dc72a867c3b25b185d7cd426a438: Status 404 returned error can't find the container with id 99cd352f9f690c35ab6bcbbeac911d6699a2dc72a867c3b25b185d7cd426a438 Oct 30 15:53:12.601934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.601908 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj"] Oct 30 15:53:12.603529 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.603500 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:12.603585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.603560 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:12.603616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:12.603590 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:13.180814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.180782 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" Oct 30 15:53:13.181253 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.180815 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk" event={"ID":"eae5fda4-df24-4926-9ed7-de6b988f6ace","Type":"ContainerDied","Data":"2ffcd6df6f44a4b94d52eba685a526af132a0136431d7a193e0e2c93db47dc31"} Oct 30 15:53:13.181253 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.180900 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ffcd6df6f44a4b94d52eba685a526af132a0136431d7a193e0e2c93db47dc31" Oct 30 15:53:13.182635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.182606 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" event={"ID":"65b4e65e-4696-4b8d-a5a9-f6ab671149e8","Type":"ContainerStarted","Data":"42b4b940b771e9d1570abcb67747d9fa63092005126a250c2237758d7209dd73"} Oct 30 15:53:13.182749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.182659 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" event={"ID":"65b4e65e-4696-4b8d-a5a9-f6ab671149e8","Type":"ContainerStarted","Data":"99cd352f9f690c35ab6bcbbeac911d6699a2dc72a867c3b25b185d7cd426a438"} Oct 30 15:53:13.262323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.262253 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podStartSLOduration=2.2622340850000002 podStartE2EDuration="2.262234085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:13.248267858 +0000 UTC m=+901.947541017" watchObservedRunningTime="2026-10-30 15:53:13.262234085 +0000 UTC m=+901.961507245" Oct 30 15:53:13.265596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.265287 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:13.266341 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.266310 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:53:13.266454 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.266360 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:53:13.359848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.354669 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:13.466319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.466214 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-openshift-ca\") pod \"d5e774eb-c937-4454-842d-f83beaca68f2\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " Oct 30 15:53:13.466490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.466449 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5e774eb-c937-4454-842d-f83beaca68f2-kube-api-access-n8jsb\" (UniqueName: \"kubernetes.io/projected/d5e774eb-c937-4454-842d-f83beaca68f2-kube-api-access-n8jsb\") pod \"d5e774eb-c937-4454-842d-f83beaca68f2\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " Oct 30 15:53:13.466490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.466482 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-xks-ca\") pod \"d5e774eb-c937-4454-842d-f83beaca68f2\" (UID: \"d5e774eb-c937-4454-842d-f83beaca68f2\") " Oct 30 15:53:13.467457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.467432 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-openshift-ca" pod "d5e774eb-c937-4454-842d-f83beaca68f2" (UID: "d5e774eb-c937-4454-842d-f83beaca68f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:13.476417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.476384 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5e774eb-c937-4454-842d-f83beaca68f2-kube-api-access-n8jsb" pod "d5e774eb-c937-4454-842d-f83beaca68f2" (UID: "d5e774eb-c937-4454-842d-f83beaca68f2"). InnerVolumeSpecName "kube-api-access-n8jsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:13.477233 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.477203 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-xks-ca" pod "d5e774eb-c937-4454-842d-f83beaca68f2" (UID: "d5e774eb-c937-4454-842d-f83beaca68f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:13.569925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.569788 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:13.569925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.569858 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8jsb\" (UniqueName: \"kubernetes.io/projected/d5e774eb-c937-4454-842d-f83beaca68f2-kube-api-access-n8jsb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:13.569925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:13.569887 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5e774eb-c937-4454-842d-f83beaca68f2-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:14.191009 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.190979 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" Oct 30 15:53:14.191406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.191014 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj" event={"ID":"d5e774eb-c937-4454-842d-f83beaca68f2","Type":"ContainerDied","Data":"2e4dcd8f0f3d22ab222c62abdb710d6ddfd730ce672bf2a441cd674aeb0833ae"} Oct 30 15:53:14.191406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.191060 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e4dcd8f0f3d22ab222c62abdb710d6ddfd730ce672bf2a441cd674aeb0833ae" Oct 30 15:53:14.271279 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.271233 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:53:14.271455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.271298 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:53:14.636749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.636709 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z"] Oct 30 15:53:14.637618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.637583 4401 cpu_manager.go:414] "containerMap: removing container" podUID="eae5fda4-df24-4926-9ed7-de6b988f6ace" containerName="revoke-keys" Oct 30 15:53:14.637618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.637612 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae5fda4-df24-4926-9ed7-de6b988f6ace" containerName="revoke-keys" Oct 30 15:53:14.637993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.637656 4401 cpu_manager.go:414] "containerMap: removing container" podUID="d5e774eb-c937-4454-842d-f83beaca68f2" containerName="revoke-keys" Oct 30 15:53:14.637993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.637664 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5e774eb-c937-4454-842d-f83beaca68f2" containerName="revoke-keys" Oct 30 15:53:14.637993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.637841 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae5fda4-df24-4926-9ed7-de6b988f6ace" containerName="revoke-keys" Oct 30 15:53:14.637993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.637865 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5e774eb-c937-4454-842d-f83beaca68f2" containerName="revoke-keys" Oct 30 15:53:14.658693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.658664 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.695871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.691928 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:53:14.746874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.698070 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:53:14.746874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.698894 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.746874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.745804 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.746874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.745986 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8trlc\" (UniqueName: \"kubernetes.io/projected/9bb0b022-90b9-4e1c-b305-347a66137d60-kube-api-access-8trlc\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.746874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.745492 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z"] Oct 30 15:53:14.746874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.707060 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:53:14.746874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.707104 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:53:14.871597 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.871563 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.871853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.871621 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.871853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.871723 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8trlc\" (UniqueName: \"kubernetes.io/projected/9bb0b022-90b9-4e1c-b305-347a66137d60-kube-api-access-8trlc\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.872585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.872551 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.875082 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.875056 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.879895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.879866 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz"] Oct 30 15:53:14.884528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.884509 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:14.899746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.899723 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8trlc\" (UniqueName: \"kubernetes.io/projected/9bb0b022-90b9-4e1c-b305-347a66137d60-kube-api-access-8trlc\") pod \"maas-api-revoke-sub-15068d95-frh9z\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:14.905540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:14.905509 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz"] Oct 30 15:53:15.023886 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.023812 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:15.074669 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.073792 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvpfv\" (UniqueName: \"kubernetes.io/projected/e74e579a-5a54-4e78-9035-0bd06d600337-kube-api-access-fvpfv\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.074669 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.073962 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.074669 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.074057 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.177886 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.177748 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvpfv\" (UniqueName: \"kubernetes.io/projected/e74e579a-5a54-4e78-9035-0bd06d600337-kube-api-access-fvpfv\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.178770 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.178267 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.179049 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.179028 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.180018 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.179903 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.181212 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.181194 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.203357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.203332 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvpfv\" (UniqueName: \"kubernetes.io/projected/e74e579a-5a54-4e78-9035-0bd06d600337-kube-api-access-fvpfv\") pod \"maas-api-revoke-sub-7b0d063d-vbgsz\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.267634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.267183 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:53:15.267634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.267243 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:53:15.373241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.373051 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z"] Oct 30 15:53:15.376817 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:15.376790 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb0b022_90b9_4e1c_b305_347a66137d60.slice/crio-9e892f7504a678b3c2e2e7724f557660ba28ccfa0731df3196ef7b8f6b601fec WatchSource:0}: Error finding container 9e892f7504a678b3c2e2e7724f557660ba28ccfa0731df3196ef7b8f6b601fec: Status 404 returned error can't find the container with id 9e892f7504a678b3c2e2e7724f557660ba28ccfa0731df3196ef7b8f6b601fec Oct 30 15:53:15.499628 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.499545 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:15.839524 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:15.839494 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74e579a_5a54_4e78_9035_0bd06d600337.slice/crio-6312cb006f41d18541d73205dfdb21ed396ec09436f70edbb1da2828bc672b14 WatchSource:0}: Error finding container 6312cb006f41d18541d73205dfdb21ed396ec09436f70edbb1da2828bc672b14: Status 404 returned error can't find the container with id 6312cb006f41d18541d73205dfdb21ed396ec09436f70edbb1da2828bc672b14 Oct 30 15:53:15.840069 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:15.840025 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz"] Oct 30 15:53:16.206920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:16.206870 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" event={"ID":"9bb0b022-90b9-4e1c-b305-347a66137d60","Type":"ContainerStarted","Data":"a290eecbab58d210cf975654525d2864bba23ba214755152ae178eab50ea5ee3"} Oct 30 15:53:16.207355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:16.206934 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" event={"ID":"9bb0b022-90b9-4e1c-b305-347a66137d60","Type":"ContainerStarted","Data":"9e892f7504a678b3c2e2e7724f557660ba28ccfa0731df3196ef7b8f6b601fec"} Oct 30 15:53:16.208360 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:16.208331 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" event={"ID":"e74e579a-5a54-4e78-9035-0bd06d600337","Type":"ContainerStarted","Data":"6312cb006f41d18541d73205dfdb21ed396ec09436f70edbb1da2828bc672b14"} Oct 30 15:53:16.270849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:16.264909 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" podStartSLOduration=2.264890568 podStartE2EDuration="2.264890568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:16.26330129 +0000 UTC m=+904.962574448" watchObservedRunningTime="2026-10-30 15:53:16.264890568 +0000 UTC m=+904.964163727" Oct 30 15:53:16.274001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:16.273958 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:53:16.274179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:16.274019 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:53:17.217473 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:17.217434 4401 generic.go:356] "Generic (PLEG): container finished" podID="9bb0b022-90b9-4e1c-b305-347a66137d60" containerID="a290eecbab58d210cf975654525d2864bba23ba214755152ae178eab50ea5ee3" exitCode=0 Oct 30 15:53:17.217923 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:17.217523 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" event={"ID":"9bb0b022-90b9-4e1c-b305-347a66137d60","Type":"ContainerDied","Data":"a290eecbab58d210cf975654525d2864bba23ba214755152ae178eab50ea5ee3"} Oct 30 15:53:17.219193 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:17.219170 4401 generic.go:356] "Generic (PLEG): container finished" podID="e74e579a-5a54-4e78-9035-0bd06d600337" containerID="af69c054864ef9553a3d4b7d5065a7974fb57d80fa564d3bb04069b5d1e795ef" exitCode=0 Oct 30 15:53:17.219296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:17.219252 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" event={"ID":"e74e579a-5a54-4e78-9035-0bd06d600337","Type":"ContainerDied","Data":"af69c054864ef9553a3d4b7d5065a7974fb57d80fa564d3bb04069b5d1e795ef"} Oct 30 15:53:17.267903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:17.265925 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:53:17.267903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:17.265986 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:53:18.274762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.274290 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:53:18.274762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.274363 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:53:18.495441 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.494291 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:18.552501 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.545445 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74e579a-5a54-4e78-9035-0bd06d600337-kube-api-access-fvpfv\" (UniqueName: \"kubernetes.io/projected/e74e579a-5a54-4e78-9035-0bd06d600337-kube-api-access-fvpfv\") pod \"e74e579a-5a54-4e78-9035-0bd06d600337\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " Oct 30 15:53:18.552501 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.545654 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-xks-ca\") pod \"e74e579a-5a54-4e78-9035-0bd06d600337\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " Oct 30 15:53:18.552501 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.545690 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-openshift-ca\") pod \"e74e579a-5a54-4e78-9035-0bd06d600337\" (UID: \"e74e579a-5a54-4e78-9035-0bd06d600337\") " Oct 30 15:53:18.557598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.550201 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-openshift-ca" pod "e74e579a-5a54-4e78-9035-0bd06d600337" (UID: "e74e579a-5a54-4e78-9035-0bd06d600337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:18.557598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.556007 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74e579a-5a54-4e78-9035-0bd06d600337-kube-api-access-fvpfv" pod "e74e579a-5a54-4e78-9035-0bd06d600337" (UID: "e74e579a-5a54-4e78-9035-0bd06d600337"). InnerVolumeSpecName "kube-api-access-fvpfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:18.581030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.580982 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-xks-ca" pod "e74e579a-5a54-4e78-9035-0bd06d600337" (UID: "e74e579a-5a54-4e78-9035-0bd06d600337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:18.581740 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.581711 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvpfv\" (UniqueName: \"kubernetes.io/projected/e74e579a-5a54-4e78-9035-0bd06d600337-kube-api-access-fvpfv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:18.581792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.581750 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:18.581792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.581769 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e74e579a-5a54-4e78-9035-0bd06d600337-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:18.593299 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.593275 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:18.683385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.683350 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-openshift-ca\") pod \"9bb0b022-90b9-4e1c-b305-347a66137d60\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " Oct 30 15:53:18.683595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.683455 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb0b022-90b9-4e1c-b305-347a66137d60-kube-api-access-8trlc\" (UniqueName: \"kubernetes.io/projected/9bb0b022-90b9-4e1c-b305-347a66137d60-kube-api-access-8trlc\") pod \"9bb0b022-90b9-4e1c-b305-347a66137d60\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " Oct 30 15:53:18.683669 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.683599 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-xks-ca\") pod \"9bb0b022-90b9-4e1c-b305-347a66137d60\" (UID: \"9bb0b022-90b9-4e1c-b305-347a66137d60\") " Oct 30 15:53:18.687324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.685288 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-openshift-ca" pod "9bb0b022-90b9-4e1c-b305-347a66137d60" (UID: "9bb0b022-90b9-4e1c-b305-347a66137d60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:18.687324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.687254 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-xks-ca" pod "9bb0b022-90b9-4e1c-b305-347a66137d60" (UID: "9bb0b022-90b9-4e1c-b305-347a66137d60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:18.691590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.691568 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb0b022-90b9-4e1c-b305-347a66137d60-kube-api-access-8trlc" pod "9bb0b022-90b9-4e1c-b305-347a66137d60" (UID: "9bb0b022-90b9-4e1c-b305-347a66137d60"). InnerVolumeSpecName "kube-api-access-8trlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:18.785557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.785472 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8trlc\" (UniqueName: \"kubernetes.io/projected/9bb0b022-90b9-4e1c-b305-347a66137d60-kube-api-access-8trlc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:18.785557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.785506 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:18.785557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.785518 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bb0b022-90b9-4e1c-b305-347a66137d60-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:18.807960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.807926 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67"] Oct 30 15:53:18.808533 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.808519 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e74e579a-5a54-4e78-9035-0bd06d600337" containerName="revoke-keys" Oct 30 15:53:18.808576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.808534 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74e579a-5a54-4e78-9035-0bd06d600337" containerName="revoke-keys" Oct 30 15:53:18.808612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.808575 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9bb0b022-90b9-4e1c-b305-347a66137d60" containerName="revoke-keys" Oct 30 15:53:18.808612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.808581 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb0b022-90b9-4e1c-b305-347a66137d60" containerName="revoke-keys" Oct 30 15:53:18.808684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.808674 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb0b022-90b9-4e1c-b305-347a66137d60" containerName="revoke-keys" Oct 30 15:53:18.808715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.808689 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74e579a-5a54-4e78-9035-0bd06d600337" containerName="revoke-keys" Oct 30 15:53:18.815223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.815186 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.831535 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.831499 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67"] Oct 30 15:53:18.886205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.886170 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.886368 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.886293 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.886415 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.886396 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fg5d\" (UniqueName: \"kubernetes.io/projected/adfebbd8-8b40-4915-bac2-73284dea8d20-kube-api-access-6fg5d\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.988413 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.988372 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fg5d\" (UniqueName: \"kubernetes.io/projected/adfebbd8-8b40-4915-bac2-73284dea8d20-kube-api-access-6fg5d\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.988561 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.988520 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.988651 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.988633 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.989202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.989173 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:18.990911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:18.990892 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:19.061539 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.061463 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fg5d\" (UniqueName: \"kubernetes.io/projected/adfebbd8-8b40-4915-bac2-73284dea8d20-kube-api-access-6fg5d\") pod \"maas-api-revoke-sub-4313ab02-5hs67\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:19.140371 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.140338 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:19.236720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.236691 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" Oct 30 15:53:19.236895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.236728 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z" event={"ID":"9bb0b022-90b9-4e1c-b305-347a66137d60","Type":"ContainerDied","Data":"9e892f7504a678b3c2e2e7724f557660ba28ccfa0731df3196ef7b8f6b601fec"} Oct 30 15:53:19.236895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.236775 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e892f7504a678b3c2e2e7724f557660ba28ccfa0731df3196ef7b8f6b601fec" Oct 30 15:53:19.238397 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.238376 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" Oct 30 15:53:19.238483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.238373 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz" event={"ID":"e74e579a-5a54-4e78-9035-0bd06d600337","Type":"ContainerDied","Data":"6312cb006f41d18541d73205dfdb21ed396ec09436f70edbb1da2828bc672b14"} Oct 30 15:53:19.238521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.238507 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6312cb006f41d18541d73205dfdb21ed396ec09436f70edbb1da2828bc672b14" Oct 30 15:53:19.264401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.264363 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:53:19.264542 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.264415 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 15:53:19.453507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.453479 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67"] Oct 30 15:53:19.453812 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:19.453510 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfebbd8_8b40_4915_bac2_73284dea8d20.slice/crio-657956aa9e8cdfb443862e714375830314a761333668e859481d11ecd8fc7119 WatchSource:0}: Error finding container 657956aa9e8cdfb443862e714375830314a761333668e859481d11ecd8fc7119: Status 404 returned error can't find the container with id 657956aa9e8cdfb443862e714375830314a761333668e859481d11ecd8fc7119 Oct 30 15:53:19.538589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.538553 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87"] Oct 30 15:53:19.545339 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.545310 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.563117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.563079 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87"] Oct 30 15:53:19.698942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.698909 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcfkg\" (UniqueName: \"kubernetes.io/projected/945701ed-1b70-46d9-b8a9-ef9605c0f068-kube-api-access-wcfkg\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.699124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.699048 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.699168 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.699122 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.800417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.800382 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcfkg\" (UniqueName: \"kubernetes.io/projected/945701ed-1b70-46d9-b8a9-ef9605c0f068-kube-api-access-wcfkg\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.800580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.800476 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.800580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.800518 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.801173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.801146 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.803010 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.802988 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.825383 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.825360 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcfkg\" (UniqueName: \"kubernetes.io/projected/945701ed-1b70-46d9-b8a9-ef9605c0f068-kube-api-access-wcfkg\") pod \"maas-api-revoke-sub-a9c4181b-kgs87\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:19.866045 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:19.866016 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:20.170410 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:20.170381 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945701ed_1b70_46d9_b8a9_ef9605c0f068.slice/crio-e1e9743aed9b0ca7973f92806d99c7bbbf6b5f90c73c4643d422007fd52bfd10 WatchSource:0}: Error finding container e1e9743aed9b0ca7973f92806d99c7bbbf6b5f90c73c4643d422007fd52bfd10: Status 404 returned error can't find the container with id e1e9743aed9b0ca7973f92806d99c7bbbf6b5f90c73c4643d422007fd52bfd10 Oct 30 15:53:20.171017 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.170996 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87"] Oct 30 15:53:20.252163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.252124 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" event={"ID":"adfebbd8-8b40-4915-bac2-73284dea8d20","Type":"ContainerStarted","Data":"4776eb456a670d60d34cec27848522b18f626cb86ab6fffce60419c4665e0ed4"} Oct 30 15:53:20.252163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.252190 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" event={"ID":"adfebbd8-8b40-4915-bac2-73284dea8d20","Type":"ContainerStarted","Data":"657956aa9e8cdfb443862e714375830314a761333668e859481d11ecd8fc7119"} Oct 30 15:53:20.254369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.254339 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" event={"ID":"945701ed-1b70-46d9-b8a9-ef9605c0f068","Type":"ContainerStarted","Data":"e1e9743aed9b0ca7973f92806d99c7bbbf6b5f90c73c4643d422007fd52bfd10"} Oct 30 15:53:20.268489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.268461 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:20.268764 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.268746 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:20.269530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.269512 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:53:20.301443 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:20.301394 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" podStartSLOduration=2.3013801369999998 podStartE2EDuration="2.301380137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:20.297117818 +0000 UTC m=+908.996390987" watchObservedRunningTime="2026-10-30 15:53:20.301380137 +0000 UTC m=+909.000653307" Oct 30 15:53:21.263617 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:21.263586 4401 generic.go:356] "Generic (PLEG): container finished" podID="adfebbd8-8b40-4915-bac2-73284dea8d20" containerID="4776eb456a670d60d34cec27848522b18f626cb86ab6fffce60419c4665e0ed4" exitCode=0 Oct 30 15:53:21.264041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:21.263661 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" event={"ID":"adfebbd8-8b40-4915-bac2-73284dea8d20","Type":"ContainerDied","Data":"4776eb456a670d60d34cec27848522b18f626cb86ab6fffce60419c4665e0ed4"} Oct 30 15:53:21.265321 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:21.265297 4401 generic.go:356] "Generic (PLEG): container finished" podID="945701ed-1b70-46d9-b8a9-ef9605c0f068" containerID="ab326fe6ee9f9ab38a8e2f84745d0b69d173f83b8de9683cec68f8f09821f259" exitCode=0 Oct 30 15:53:21.265435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:21.265349 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" event={"ID":"945701ed-1b70-46d9-b8a9-ef9605c0f068","Type":"ContainerDied","Data":"ab326fe6ee9f9ab38a8e2f84745d0b69d173f83b8de9683cec68f8f09821f259"} Oct 30 15:53:21.967776 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:53:21.967735 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:53:22.474434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.474406 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:22.476719 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.476700 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:22.661096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661067 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-openshift-ca\") pod \"adfebbd8-8b40-4915-bac2-73284dea8d20\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " Oct 30 15:53:22.661289 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661142 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfebbd8-8b40-4915-bac2-73284dea8d20-kube-api-access-6fg5d\" (UniqueName: \"kubernetes.io/projected/adfebbd8-8b40-4915-bac2-73284dea8d20-kube-api-access-6fg5d\") pod \"adfebbd8-8b40-4915-bac2-73284dea8d20\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " Oct 30 15:53:22.661289 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661178 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-xks-ca\") pod \"945701ed-1b70-46d9-b8a9-ef9605c0f068\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " Oct 30 15:53:22.661289 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661235 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945701ed-1b70-46d9-b8a9-ef9605c0f068-kube-api-access-wcfkg\" (UniqueName: \"kubernetes.io/projected/945701ed-1b70-46d9-b8a9-ef9605c0f068-kube-api-access-wcfkg\") pod \"945701ed-1b70-46d9-b8a9-ef9605c0f068\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " Oct 30 15:53:22.661464 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661356 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-openshift-ca\") pod \"945701ed-1b70-46d9-b8a9-ef9605c0f068\" (UID: \"945701ed-1b70-46d9-b8a9-ef9605c0f068\") " Oct 30 15:53:22.661464 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661409 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-openshift-ca" pod "adfebbd8-8b40-4915-bac2-73284dea8d20" (UID: "adfebbd8-8b40-4915-bac2-73284dea8d20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:22.661567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661469 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-xks-ca\") pod \"adfebbd8-8b40-4915-bac2-73284dea8d20\" (UID: \"adfebbd8-8b40-4915-bac2-73284dea8d20\") " Oct 30 15:53:22.661739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.661713 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-openshift-ca" pod "945701ed-1b70-46d9-b8a9-ef9605c0f068" (UID: "945701ed-1b70-46d9-b8a9-ef9605c0f068"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:22.662028 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.662007 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:22.662144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.662030 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:22.663593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.663573 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfebbd8-8b40-4915-bac2-73284dea8d20-kube-api-access-6fg5d" pod "adfebbd8-8b40-4915-bac2-73284dea8d20" (UID: "adfebbd8-8b40-4915-bac2-73284dea8d20"). InnerVolumeSpecName "kube-api-access-6fg5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:22.663702 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.663599 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945701ed-1b70-46d9-b8a9-ef9605c0f068-kube-api-access-wcfkg" pod "945701ed-1b70-46d9-b8a9-ef9605c0f068" (UID: "945701ed-1b70-46d9-b8a9-ef9605c0f068"). InnerVolumeSpecName "kube-api-access-wcfkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:22.663771 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.663696 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-xks-ca" pod "945701ed-1b70-46d9-b8a9-ef9605c0f068" (UID: "945701ed-1b70-46d9-b8a9-ef9605c0f068"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:22.663850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.663771 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-xks-ca" pod "adfebbd8-8b40-4915-bac2-73284dea8d20" (UID: "adfebbd8-8b40-4915-bac2-73284dea8d20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:22.763722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.763683 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fg5d\" (UniqueName: \"kubernetes.io/projected/adfebbd8-8b40-4915-bac2-73284dea8d20-kube-api-access-6fg5d\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:22.763722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.763717 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/945701ed-1b70-46d9-b8a9-ef9605c0f068-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:22.763722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.763727 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcfkg\" (UniqueName: \"kubernetes.io/projected/945701ed-1b70-46d9-b8a9-ef9605c0f068-kube-api-access-wcfkg\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:22.763987 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:22.763735 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adfebbd8-8b40-4915-bac2-73284dea8d20-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:23.295353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:23.295322 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" Oct 30 15:53:23.295546 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:23.295322 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67" event={"ID":"adfebbd8-8b40-4915-bac2-73284dea8d20","Type":"ContainerDied","Data":"657956aa9e8cdfb443862e714375830314a761333668e859481d11ecd8fc7119"} Oct 30 15:53:23.295546 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:23.295446 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="657956aa9e8cdfb443862e714375830314a761333668e859481d11ecd8fc7119" Oct 30 15:53:23.296783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:23.296758 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" Oct 30 15:53:23.296935 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:23.296798 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87" event={"ID":"945701ed-1b70-46d9-b8a9-ef9605c0f068","Type":"ContainerDied","Data":"e1e9743aed9b0ca7973f92806d99c7bbbf6b5f90c73c4643d422007fd52bfd10"} Oct 30 15:53:23.296935 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:23.296872 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1e9743aed9b0ca7973f92806d99c7bbbf6b5f90c73c4643d422007fd52bfd10" Oct 30 15:53:23.747207 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:23.747174 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:53:24.744011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.743970 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw"] Oct 30 15:53:24.745215 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.745186 4401 cpu_manager.go:414] "containerMap: removing container" podUID="945701ed-1b70-46d9-b8a9-ef9605c0f068" containerName="revoke-keys" Oct 30 15:53:24.745384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.745370 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945701ed-1b70-46d9-b8a9-ef9605c0f068" containerName="revoke-keys" Oct 30 15:53:24.745537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.745524 4401 cpu_manager.go:414] "containerMap: removing container" podUID="adfebbd8-8b40-4915-bac2-73284dea8d20" containerName="revoke-keys" Oct 30 15:53:24.745619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.745608 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfebbd8-8b40-4915-bac2-73284dea8d20" containerName="revoke-keys" Oct 30 15:53:24.745879 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.745862 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="945701ed-1b70-46d9-b8a9-ef9605c0f068" containerName="revoke-keys" Oct 30 15:53:24.746005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.745993 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfebbd8-8b40-4915-bac2-73284dea8d20" containerName="revoke-keys" Oct 30 15:53:24.758653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.758626 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:24.778214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.775490 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw"] Oct 30 15:53:24.782961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:24.782760 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:53:25.124910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.046211 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:53:25.124910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.046344 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:53:25.124910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.046443 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:53:25.124910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.051000 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.124910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.051140 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.124910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.051198 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtmm\" (UniqueName: \"kubernetes.io/projected/d8b81e87-50cb-416b-8670-59c6329084e1-kube-api-access-6jtmm\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.206848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.172679 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.206848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.189885 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.206848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.189988 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtmm\" (UniqueName: \"kubernetes.io/projected/d8b81e87-50cb-416b-8670-59c6329084e1-kube-api-access-6jtmm\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.314848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.314047 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.382974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.360943 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.382974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.371412 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtmm\" (UniqueName: \"kubernetes.io/projected/d8b81e87-50cb-416b-8670-59c6329084e1-kube-api-access-6jtmm\") pod \"maas-api-revoke-sub-539d3120-gscxw\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:25.491091 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:25.491058 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:26.367857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:26.365914 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:26.389975 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:26.387066 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:26.672637 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:26.670294 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:26.672637 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:26.672083 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:53:27.005994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:26.973671 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:53:27.047767 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.047722 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:27.390925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.367357 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:27.390925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.371431 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:53:27.392331 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.391582 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:27.393404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.392566 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:53:27.608848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.597383 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw"] Oct 30 15:53:27.670845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.670732 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:27.822724 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.711455 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" event={"ID":"d8b81e87-50cb-416b-8670-59c6329084e1","Type":"ContainerStarted","Data":"c39a96a32d976580e185493e5bd6a23b9b6e0b2584349527b0dbdacb25495ccf"} Oct 30 15:53:27.823591 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.823565 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:53:27.989857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:27.984489 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:53:28.003293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.003262 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:53:28.064999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.064967 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:53:28.380850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.377841 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:53:28.440492 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.406081 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:53:28.665300 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.636283 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:53:28.725843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.723869 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:53:28.845852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.841625 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj"] Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.893282 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj"] Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.893476 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.911432 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-c129d1\"/\"distinct-w6-c129d1-kserve-self-signed-certs\"" Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.916535 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-home\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.916713 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tmp-dir\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.916819 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-dshm\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.916885 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-model-cache\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.917013 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg7px\" (UniqueName: \"kubernetes.io/projected/0ca1b88e-6358-49af-bcc8-1eba20dd115a-kube-api-access-cg7px\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:28.922903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:28.917103 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tls-certs\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.032027 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.029927 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-home\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.036269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.036236 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tmp-dir\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.042143 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.042106 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-dshm\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.042420 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.042398 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-model-cache\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.042600 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.042580 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg7px\" (UniqueName: \"kubernetes.io/projected/0ca1b88e-6358-49af-bcc8-1eba20dd115a-kube-api-access-cg7px\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.044908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.044884 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tls-certs\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.074846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.033117 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-home\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.076014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.066102 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-model-cache\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.127849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.127158 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tmp-dir\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.127849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.127363 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-dshm\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.178267 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.176294 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg7px\" (UniqueName: \"kubernetes.io/projected/0ca1b88e-6358-49af-bcc8-1eba20dd115a-kube-api-access-cg7px\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.231785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.231745 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tls-certs\") pod \"distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.421266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.421011 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:29.941379 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:29.941350 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ca1b88e_6358_49af_bcc8_1eba20dd115a.slice/crio-d2db846026e174793db5a27a3b771f32272012955a658f23e8618e226c3d1f10 WatchSource:0}: Error finding container d2db846026e174793db5a27a3b771f32272012955a658f23e8618e226c3d1f10: Status 404 returned error can't find the container with id d2db846026e174793db5a27a3b771f32272012955a658f23e8618e226c3d1f10 Oct 30 15:53:29.942267 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:29.942237 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj"] Oct 30 15:53:30.181872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:30.181834 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" event={"ID":"0ca1b88e-6358-49af-bcc8-1eba20dd115a","Type":"ContainerStarted","Data":"97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae"} Oct 30 15:53:30.182051 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:30.181880 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" event={"ID":"0ca1b88e-6358-49af-bcc8-1eba20dd115a","Type":"ContainerStarted","Data":"d2db846026e174793db5a27a3b771f32272012955a658f23e8618e226c3d1f10"} Oct 30 15:53:30.183441 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:30.183411 4401 generic.go:356] "Generic (PLEG): container finished" podID="d8b81e87-50cb-416b-8670-59c6329084e1" containerID="9b54bbca16a935dee95d9c96a8e072ee32bae1e62b84a2493554f06d7dadb368" exitCode=0 Oct 30 15:53:30.183553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:30.183487 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" event={"ID":"d8b81e87-50cb-416b-8670-59c6329084e1","Type":"ContainerDied","Data":"9b54bbca16a935dee95d9c96a8e072ee32bae1e62b84a2493554f06d7dadb368"} Oct 30 15:53:30.229914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:30.229787 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podStartSLOduration=2.229774237 podStartE2EDuration="2.229774237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:30.226650001 +0000 UTC m=+918.925923158" watchObservedRunningTime="2026-10-30 15:53:30.229774237 +0000 UTC m=+918.929047388" Oct 30 15:53:30.422564 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:30.422535 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:31.419701 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.419674 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:31.508640 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.508564 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-openshift-ca\") pod \"d8b81e87-50cb-416b-8670-59c6329084e1\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " Oct 30 15:53:31.508793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.508640 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-xks-ca\") pod \"d8b81e87-50cb-416b-8670-59c6329084e1\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " Oct 30 15:53:31.508793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.508673 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b81e87-50cb-416b-8670-59c6329084e1-kube-api-access-6jtmm\" (UniqueName: \"kubernetes.io/projected/d8b81e87-50cb-416b-8670-59c6329084e1-kube-api-access-6jtmm\") pod \"d8b81e87-50cb-416b-8670-59c6329084e1\" (UID: \"d8b81e87-50cb-416b-8670-59c6329084e1\") " Oct 30 15:53:31.508970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.508922 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-openshift-ca" pod "d8b81e87-50cb-416b-8670-59c6329084e1" (UID: "d8b81e87-50cb-416b-8670-59c6329084e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:31.509276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.509250 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:31.510792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.510770 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b81e87-50cb-416b-8670-59c6329084e1-kube-api-access-6jtmm" pod "d8b81e87-50cb-416b-8670-59c6329084e1" (UID: "d8b81e87-50cb-416b-8670-59c6329084e1"). InnerVolumeSpecName "kube-api-access-6jtmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:31.510980 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.510791 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-xks-ca" pod "d8b81e87-50cb-416b-8670-59c6329084e1" (UID: "d8b81e87-50cb-416b-8670-59c6329084e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:31.610886 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.610851 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8b81e87-50cb-416b-8670-59c6329084e1-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:31.610886 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:31.610880 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jtmm\" (UniqueName: \"kubernetes.io/projected/d8b81e87-50cb-416b-8670-59c6329084e1-kube-api-access-6jtmm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:32.203476 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:32.203441 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" Oct 30 15:53:32.203476 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:32.203454 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw" event={"ID":"d8b81e87-50cb-416b-8670-59c6329084e1","Type":"ContainerDied","Data":"c39a96a32d976580e185493e5bd6a23b9b6e0b2584349527b0dbdacb25495ccf"} Oct 30 15:53:32.203678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:32.203504 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c39a96a32d976580e185493e5bd6a23b9b6e0b2584349527b0dbdacb25495ccf" Oct 30 15:53:35.064575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.064529 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg"] Oct 30 15:53:35.065196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.065181 4401 cpu_manager.go:414] "containerMap: removing container" podUID="d8b81e87-50cb-416b-8670-59c6329084e1" containerName="revoke-keys" Oct 30 15:53:35.065196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.065196 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b81e87-50cb-416b-8670-59c6329084e1" containerName="revoke-keys" Oct 30 15:53:35.065301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.065292 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b81e87-50cb-416b-8670-59c6329084e1" containerName="revoke-keys" Oct 30 15:53:35.068783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.068766 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.086668 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.086641 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg"] Oct 30 15:53:35.095086 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.095061 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:53:35.095218 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.095091 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:53:35.095218 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.095115 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:53:35.103435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.103417 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:53:35.161058 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.161031 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8pl4\" (UniqueName: \"kubernetes.io/projected/b649fdd2-99c9-4a74-9da8-c4d872965e1c-kube-api-access-h8pl4\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.161195 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.161079 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.161195 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.161179 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.262616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.262583 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8pl4\" (UniqueName: \"kubernetes.io/projected/b649fdd2-99c9-4a74-9da8-c4d872965e1c-kube-api-access-h8pl4\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.262810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.262638 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.262810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.262681 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.263574 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.263548 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.265545 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.265519 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.287585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.287558 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8pl4\" (UniqueName: \"kubernetes.io/projected/b649fdd2-99c9-4a74-9da8-c4d872965e1c-kube-api-access-h8pl4\") pod \"maas-api-revoke-sub-31fe52ec-lnmvg\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.383584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.383555 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:35.690141 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:35.690116 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb649fdd2_99c9_4a74_9da8_c4d872965e1c.slice/crio-fe85b82885414e7c5a9a4eaa0fdd0affba74c549c8595d4a443ef9b058b0b69c WatchSource:0}: Error finding container fe85b82885414e7c5a9a4eaa0fdd0affba74c549c8595d4a443ef9b058b0b69c: Status 404 returned error can't find the container with id fe85b82885414e7c5a9a4eaa0fdd0affba74c549c8595d4a443ef9b058b0b69c Oct 30 15:53:35.691305 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:35.691282 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg"] Oct 30 15:53:36.234323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:36.234214 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" event={"ID":"b649fdd2-99c9-4a74-9da8-c4d872965e1c","Type":"ContainerStarted","Data":"fcf6c9466238f4084fb55922d1229bf5bdacfcec8fe430b08e8f60ea9c696109"} Oct 30 15:53:36.234323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:36.234297 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" event={"ID":"b649fdd2-99c9-4a74-9da8-c4d872965e1c","Type":"ContainerStarted","Data":"fe85b82885414e7c5a9a4eaa0fdd0affba74c549c8595d4a443ef9b058b0b69c"} Oct 30 15:53:36.279259 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:36.279216 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" podStartSLOduration=1.279203896 podStartE2EDuration="1.279203896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:36.278673646 +0000 UTC m=+924.977946803" watchObservedRunningTime="2026-10-30 15:53:36.279203896 +0000 UTC m=+924.978477053" Oct 30 15:53:37.243097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:37.243063 4401 generic.go:356] "Generic (PLEG): container finished" podID="b649fdd2-99c9-4a74-9da8-c4d872965e1c" containerID="fcf6c9466238f4084fb55922d1229bf5bdacfcec8fe430b08e8f60ea9c696109" exitCode=0 Oct 30 15:53:37.243487 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:37.243152 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" event={"ID":"b649fdd2-99c9-4a74-9da8-c4d872965e1c","Type":"ContainerDied","Data":"fcf6c9466238f4084fb55922d1229bf5bdacfcec8fe430b08e8f60ea9c696109"} Oct 30 15:53:38.457016 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.456990 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:38.503679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.503651 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b649fdd2-99c9-4a74-9da8-c4d872965e1c-kube-api-access-h8pl4\" (UniqueName: \"kubernetes.io/projected/b649fdd2-99c9-4a74-9da8-c4d872965e1c-kube-api-access-h8pl4\") pod \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " Oct 30 15:53:38.503818 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.503688 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-xks-ca\") pod \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " Oct 30 15:53:38.503898 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.503871 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-openshift-ca\") pod \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\" (UID: \"b649fdd2-99c9-4a74-9da8-c4d872965e1c\") " Oct 30 15:53:38.504243 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.504214 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-openshift-ca" pod "b649fdd2-99c9-4a74-9da8-c4d872965e1c" (UID: "b649fdd2-99c9-4a74-9da8-c4d872965e1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:38.504781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.504626 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:38.506271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.506244 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-xks-ca" pod "b649fdd2-99c9-4a74-9da8-c4d872965e1c" (UID: "b649fdd2-99c9-4a74-9da8-c4d872965e1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:38.506492 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.506465 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b649fdd2-99c9-4a74-9da8-c4d872965e1c-kube-api-access-h8pl4" pod "b649fdd2-99c9-4a74-9da8-c4d872965e1c" (UID: "b649fdd2-99c9-4a74-9da8-c4d872965e1c"). InnerVolumeSpecName "kube-api-access-h8pl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:38.605961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.605933 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8pl4\" (UniqueName: \"kubernetes.io/projected/b649fdd2-99c9-4a74-9da8-c4d872965e1c-kube-api-access-h8pl4\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:38.605961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:38.605958 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b649fdd2-99c9-4a74-9da8-c4d872965e1c-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:53:39.299195 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:39.299167 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" Oct 30 15:53:39.299363 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:39.299171 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg" event={"ID":"b649fdd2-99c9-4a74-9da8-c4d872965e1c","Type":"ContainerDied","Data":"fe85b82885414e7c5a9a4eaa0fdd0affba74c549c8595d4a443ef9b058b0b69c"} Oct 30 15:53:39.299363 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:39.299307 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe85b82885414e7c5a9a4eaa0fdd0affba74c549c8595d4a443ef9b058b0b69c" Oct 30 15:53:39.429106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:39.429080 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:39.434236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:39.434213 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:40.319125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:40.319093 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:53:42.131882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.131846 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5"] Oct 30 15:53:42.134945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.134916 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b649fdd2-99c9-4a74-9da8-c4d872965e1c" containerName="revoke-keys" Oct 30 15:53:42.135126 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.135110 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b649fdd2-99c9-4a74-9da8-c4d872965e1c" containerName="revoke-keys" Oct 30 15:53:42.135463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.135446 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b649fdd2-99c9-4a74-9da8-c4d872965e1c" containerName="revoke-keys" Oct 30 15:53:42.147104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.147076 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.243951 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.243884 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5"] Oct 30 15:53:42.275174 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.275128 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.275557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.275532 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.284845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.275738 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.286218 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.286186 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.286523 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.286502 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.301846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.286739 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.303075 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.303031 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.321469 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.303339 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.447932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.443665 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzb9\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-kube-api-access-sqzb9\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550002 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550076 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550121 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzb9\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-kube-api-access-sqzb9\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550209 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550274 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550328 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550379 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550454 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.550848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.550534 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.551492 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.551022 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.574848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.552640 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.574848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.553381 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.574848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.553620 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.574848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.554116 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.602134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.602099 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.640524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.640482 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.642168 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.642053 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.713059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.712599 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzb9\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-kube-api-access-sqzb9\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:42.791046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:42.791009 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:44.011922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.011883 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5"] Oct 30 15:53:44.034856 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:44.031922 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0728c928_7b7d_4ed6_a3df_5e35727c71b0.slice/crio-9c33f35c4a3d40ba88c87d8cf800d6befdfa17ce7f33f5fe3f73af90fad19e60 WatchSource:0}: Error finding container 9c33f35c4a3d40ba88c87d8cf800d6befdfa17ce7f33f5fe3f73af90fad19e60: Status 404 returned error can't find the container with id 9c33f35c4a3d40ba88c87d8cf800d6befdfa17ce7f33f5fe3f73af90fad19e60 Oct 30 15:53:44.038607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.038144 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:44.038607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.038227 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:44.038607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.038272 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:44.507514 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.507472 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" event={"ID":"0728c928-7b7d-4ed6-a3df-5e35727c71b0","Type":"ContainerStarted","Data":"22bd8b511fcd0f8e046b15f2ba50745457a1ed9dc24b2eaa0a92e19aa068ef6a"} Oct 30 15:53:44.507773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.507749 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" event={"ID":"0728c928-7b7d-4ed6-a3df-5e35727c71b0","Type":"ContainerStarted","Data":"9c33f35c4a3d40ba88c87d8cf800d6befdfa17ce7f33f5fe3f73af90fad19e60"} Oct 30 15:53:44.578708 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.578646 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podStartSLOduration=2.578625407 podStartE2EDuration="2.578625407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:44.573535428 +0000 UTC m=+933.272808586" watchObservedRunningTime="2026-10-30 15:53:44.578625407 +0000 UTC m=+933.277898567" Oct 30 15:53:44.792325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:44.792242 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:45.793165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:45.793128 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 15:53:45.793530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:45.793182 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 15:53:46.792174 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:46.792136 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 15:53:46.792348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:46.792193 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 15:53:47.792345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:47.792304 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 15:53:47.792864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:47.792377 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 15:53:48.792279 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:48.792241 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 15:53:48.792496 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:48.792304 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 15:53:49.792814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:49.792778 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 15:53:49.793202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:49.792844 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 15:53:50.792205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:50.792167 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Oct 30 15:53:50.792370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:50.792223 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Oct 30 15:53:51.797320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:51.797290 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:51.797774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:51.797761 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:51.798400 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:51.798379 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:53:51.973161 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:53:51.973128 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:53:52.766764 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.766734 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn"] Oct 30 15:53:52.772603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.772585 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.804271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.804238 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn"] Oct 30 15:53:52.923559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923519 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.923810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923573 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.923810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923600 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.923810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923671 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.923810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923701 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc8085-eee2-4c42-9d59-8935888b5d65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.923810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923801 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.924014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923898 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.924014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.923970 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jkgv\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-kube-api-access-6jkgv\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:52.924014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:52.924008 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.024925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.024843 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.024925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.024903 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.024929 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.024965 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025000 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc8085-eee2-4c42-9d59-8935888b5d65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025063 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025145 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025195 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jkgv\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-kube-api-access-6jkgv\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025236 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025389 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025461 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025689 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025711 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.025932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.025876 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc8085-eee2-4c42-9d59-8935888b5d65-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.027202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.027182 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.027547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.027529 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.049738 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.049712 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.050262 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.050238 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jkgv\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-kube-api-access-6jkgv\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-mrkpn\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.088465 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.088437 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:53.416611 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:53.416580 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdc8085_eee2_4c42_9d59_8935888b5d65.slice/crio-5c152c27f50d91bd47119f7f0b32b27f788ab44f05817c1bbfc98ce2866b815c WatchSource:0}: Error finding container 5c152c27f50d91bd47119f7f0b32b27f788ab44f05817c1bbfc98ce2866b815c: Status 404 returned error can't find the container with id 5c152c27f50d91bd47119f7f0b32b27f788ab44f05817c1bbfc98ce2866b815c Oct 30 15:53:53.416731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.416608 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn"] Oct 30 15:53:53.419030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.418996 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:53.419132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.419072 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:53.419132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.419120 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:53.621350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.621319 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" event={"ID":"6cdc8085-eee2-4c42-9d59-8935888b5d65","Type":"ContainerStarted","Data":"305b5530e0c66efa37bfcacecd9b511c3702080d5fa63639065d0c13012644c9"} Oct 30 15:53:53.621534 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:53.621377 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" event={"ID":"6cdc8085-eee2-4c42-9d59-8935888b5d65","Type":"ContainerStarted","Data":"5c152c27f50d91bd47119f7f0b32b27f788ab44f05817c1bbfc98ce2866b815c"} Oct 30 15:53:54.089586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:54.089549 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:53:54.090970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:54.090948 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:53:54.091032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:54.090995 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:53:55.089350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.089313 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:53:55.089561 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.089375 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:53:55.702566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.702514 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podStartSLOduration=3.7025008379999997 podStartE2EDuration="3.702500838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:53.673100226 +0000 UTC m=+942.372373411" watchObservedRunningTime="2026-10-30 15:53:55.702500838 +0000 UTC m=+944.401773995" Oct 30 15:53:55.706064 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.706038 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm"] Oct 30 15:53:55.710781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.710764 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:55.728552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.728522 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm"] Oct 30 15:53:55.737078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.737051 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b24e7b\"/\"distinct-w4-b24e7b-kserve-self-signed-certs\"" Oct 30 15:53:55.893428 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.893390 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-kube-api-access-mc5jf\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:55.900986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.900948 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-home\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:55.901330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.901306 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tmp-dir\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:55.901528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.901511 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-dshm\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:55.901739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.901719 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-model-cache\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:55.901943 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:55.901923 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tls-certs\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.006801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.006675 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-dshm\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.007615 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.007590 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-model-cache\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.008163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.008144 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tls-certs\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.008448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.008428 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-kube-api-access-mc5jf\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.008690 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.008673 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-home\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.017849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.008795 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tmp-dir\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.018467 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.010231 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-dshm\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.018634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.008076 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-model-cache\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.018738 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.011178 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-home\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.018849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.011469 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tmp-dir\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.018957 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.018502 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tls-certs\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.046334 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.046289 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-kube-api-access-mc5jf\") pod \"distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.096798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.096759 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:53:56.097119 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.097086 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:53:56.326224 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.326143 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:56.679088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:56.679053 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm"] Oct 30 15:53:56.679229 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:56.679206 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2046f38_9cf6_4b6d_9bbf_cd5077bca9ec.slice/crio-d554c6067597e46c8f112763c3a7a3a9502394e41bc3431e8766b112bc08df14 WatchSource:0}: Error finding container d554c6067597e46c8f112763c3a7a3a9502394e41bc3431e8766b112bc08df14: Status 404 returned error can't find the container with id d554c6067597e46c8f112763c3a7a3a9502394e41bc3431e8766b112bc08df14 Oct 30 15:53:57.089223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.089127 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:53:57.089602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.089219 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:53:57.228180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.228141 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:53:57.280560 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.280529 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf"] Oct 30 15:53:57.292192 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.292155 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.305083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.305054 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf"] Oct 30 15:53:57.440394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440360 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440419 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eca30d-cb5e-417b-ae35-6faefa966211-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440527 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440564 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eca30d-cb5e-417b-ae35-6faefa966211-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440688 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440583 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440763 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440745 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440856 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5jkt\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-kube-api-access-z5jkt\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440881 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.440949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.440925 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.541918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.541877 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.541966 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5jkt\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-kube-api-access-z5jkt\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.541985 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542030 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542057 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542116 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eca30d-cb5e-417b-ae35-6faefa966211-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542181 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542218 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eca30d-cb5e-417b-ae35-6faefa966211-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542247 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542471 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542574 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542597 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.542666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.542648 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.543044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.543019 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eca30d-cb5e-417b-ae35-6faefa966211-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.544351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.544330 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.544648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.544629 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eca30d-cb5e-417b-ae35-6faefa966211-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.572901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.572865 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5jkt\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-kube-api-access-z5jkt\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.573968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.573941 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.612679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.612650 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:57.688992 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.688913 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" event={"ID":"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec","Type":"ContainerStarted","Data":"4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871"} Oct 30 15:53:57.688992 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.688964 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" event={"ID":"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec","Type":"ContainerStarted","Data":"d554c6067597e46c8f112763c3a7a3a9502394e41bc3431e8766b112bc08df14"} Oct 30 15:53:57.780852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:57.779411 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podStartSLOduration=2.779393359 podStartE2EDuration="2.779393359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:57.745357584 +0000 UTC m=+946.444630740" watchObservedRunningTime="2026-10-30 15:53:57.779393359 +0000 UTC m=+946.478666523" Oct 30 15:53:57.999106 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:53:57.999077 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eca30d_cb5e_417b_ae35_6faefa966211.slice/crio-fc23f8a84f3cf2695d8056b49c70e74c8f6ce6e1b8cc2ef4a31ebe42fa0d3143 WatchSource:0}: Error finding container fc23f8a84f3cf2695d8056b49c70e74c8f6ce6e1b8cc2ef4a31ebe42fa0d3143: Status 404 returned error can't find the container with id fc23f8a84f3cf2695d8056b49c70e74c8f6ce6e1b8cc2ef4a31ebe42fa0d3143 Oct 30 15:53:58.000251 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.000225 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf"] Oct 30 15:53:58.002054 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.002021 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:58.002122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.002090 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:58.002156 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.002120 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:53:58.088940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.088908 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:53:58.089100 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.088959 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:53:58.331930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.326926 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:53:58.735300 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.735261 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" event={"ID":"35eca30d-cb5e-417b-ae35-6faefa966211","Type":"ContainerStarted","Data":"a90b212cfcc710195550c779b29f9355ff6ff714fa8538fc6c7f3dcbdf86b014"} Oct 30 15:53:58.735587 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.735565 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" event={"ID":"35eca30d-cb5e-417b-ae35-6faefa966211","Type":"ContainerStarted","Data":"fc23f8a84f3cf2695d8056b49c70e74c8f6ce6e1b8cc2ef4a31ebe42fa0d3143"} Oct 30 15:53:58.842858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:58.842755 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podStartSLOduration=1.842734986 podStartE2EDuration="1.842734986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:58.839136029 +0000 UTC m=+947.538409188" watchObservedRunningTime="2026-10-30 15:53:58.842734986 +0000 UTC m=+947.542008144" Oct 30 15:53:59.099847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:59.096282 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:53:59.099847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:59.096346 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:53:59.620847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:59.614863 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:53:59.620847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:59.614993 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 15:53:59.620847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:53:59.615037 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 15:54:00.098622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:00.098587 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:54:00.098949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:00.098902 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:54:00.613675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:00.613642 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 15:54:00.613884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:00.613700 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 15:54:01.088851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:01.088764 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:54:01.089210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:01.088814 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:54:01.613444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:01.613405 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 15:54:01.613613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:01.613472 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 15:54:02.089744 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:02.089650 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 15:54:02.089744 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:02.089707 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 15:54:02.613135 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:02.613104 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 15:54:02.613319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:02.613159 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 15:54:03.201363 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:03.201332 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:54:03.201973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:03.201953 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:54:03.203037 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:03.203015 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:54:03.614031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:03.613996 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 15:54:03.614203 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:03.614058 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 15:54:04.613325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:04.613288 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 15:54:04.613688 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:04.613350 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 15:54:05.617171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:05.617135 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:54:05.617653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:05.617575 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:54:05.618303 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:05.618283 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:54:06.326354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:06.326315 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:54:06.339072 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:06.339038 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:54:06.886581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:06.886551 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:54:11.811168 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.811129 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b"] Oct 30 15:54:11.816575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.816555 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.828922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.828889 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.828937 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.828965 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.829001 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829197 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.829135 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6b2e0-696d-49df-81a7-f833d63e14ee-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829328 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.829304 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-549lc\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-kube-api-access-549lc\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829523 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.829494 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.829531 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.829660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.829642 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.837947 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.837915 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b"] Oct 30 15:54:11.930997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.930964 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-549lc\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-kube-api-access-549lc\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931008 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931024 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931049 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931088 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931108 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931126 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931180 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931564 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931300 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6b2e0-696d-49df-81a7-f833d63e14ee-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931564 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931432 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931667 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931567 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931667 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931617 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.931803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.931782 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.932027 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.932010 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6b2e0-696d-49df-81a7-f833d63e14ee-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.933381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.933365 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.933596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.933579 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.955909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.955884 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:11.956009 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:11.955979 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-549lc\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-kube-api-access-549lc\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:12.131780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.131744 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:12.465576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.465486 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b"] Oct 30 15:54:12.471380 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:54:12.471353 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f6b2e0_696d_49df_81a7_f833d63e14ee.slice/crio-8105aa91c049ef535e8e949b4bb719e21f1e915956b1bb3cc0d2df401f334e6d WatchSource:0}: Error finding container 8105aa91c049ef535e8e949b4bb719e21f1e915956b1bb3cc0d2df401f334e6d: Status 404 returned error can't find the container with id 8105aa91c049ef535e8e949b4bb719e21f1e915956b1bb3cc0d2df401f334e6d Oct 30 15:54:12.474087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.474055 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:54:12.474165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.474117 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:54:12.474165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.474145 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:54:12.921394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.921355 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" event={"ID":"72f6b2e0-696d-49df-81a7-f833d63e14ee","Type":"ContainerStarted","Data":"c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da"} Oct 30 15:54:12.921798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.921425 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" event={"ID":"72f6b2e0-696d-49df-81a7-f833d63e14ee","Type":"ContainerStarted","Data":"8105aa91c049ef535e8e949b4bb719e21f1e915956b1bb3cc0d2df401f334e6d"} Oct 30 15:54:12.973360 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:12.973313 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podStartSLOduration=1.9733005559999999 podStartE2EDuration="1.973300556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:12.969465233 +0000 UTC m=+961.668738390" watchObservedRunningTime="2026-10-30 15:54:12.973300556 +0000 UTC m=+961.672573775" Oct 30 15:54:13.132038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:13.132006 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:13.133403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:13.133375 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:54:13.133540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:13.133424 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 15:54:14.132869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:14.132814 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:54:14.133259 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:14.132887 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 15:54:14.818717 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:14.818686 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:54:15.132114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:15.132079 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:54:15.132285 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:15.132136 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 15:54:16.132454 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:16.132417 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:54:16.132965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:16.132476 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 15:54:17.132637 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:17.132592 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:54:17.133143 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:17.132654 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 15:54:18.137880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:18.136215 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:54:18.137880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:18.136281 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 15:54:19.132220 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:19.132177 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:54:19.132391 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:19.132251 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 15:54:20.137023 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:20.136993 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:20.137457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:20.137356 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:20.137971 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:20.137954 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:54:21.968788 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:54:21.968749 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:54:34.811150 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:34.811116 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:54:42.301526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.301495 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d"] Oct 30 15:54:42.306033 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.306013 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.351223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.351190 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d"] Oct 30 15:54:42.356559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.356531 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ph6p\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-kube-api-access-9ph6p\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.356684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.356616 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.356684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.356674 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.356789 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.356762 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.356960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.356938 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.357099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.357069 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.357224 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.357125 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.357224 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.357197 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.357369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.357345 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.458957 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.458927 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.458975 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459016 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459055 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459126 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459152 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459226 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459323 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459367 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ph6p\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-kube-api-access-9ph6p\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459431 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-certs\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459507 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-credential-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459588 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-socket\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.459806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.459780 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-data\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.460301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.460276 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istiod-ca-cert\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.461331 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.461313 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-envoy\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.461708 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.461691 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-podinfo\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.484600 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.484576 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-token\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.484700 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.484576 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ph6p\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-kube-api-access-9ph6p\") pod \"e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.634850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.634812 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:42.687442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.687410 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw"] Oct 30 15:54:42.692169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.692147 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.710970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.709421 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw"] Oct 30 15:54:42.713060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.713034 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:54:42.713283 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.713199 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:54:42.713407 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.713391 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:54:42.713550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.713534 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:54:42.762653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.762622 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.762832 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.762802 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw2nc\" (UniqueName: \"kubernetes.io/projected/238833d8-5528-4d55-8e4a-df5ee798ff55-kube-api-access-tw2nc\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.762936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.762922 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.864224 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.864187 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw2nc\" (UniqueName: \"kubernetes.io/projected/238833d8-5528-4d55-8e4a-df5ee798ff55-kube-api-access-tw2nc\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.864398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.864273 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.864398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.864341 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.865047 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.865027 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.866515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.866496 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.889567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.889504 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw2nc\" (UniqueName: \"kubernetes.io/projected/238833d8-5528-4d55-8e4a-df5ee798ff55-kube-api-access-tw2nc\") pod \"maas-api-revoke-sub-1af72af3-79xbw\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:42.987532 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:54:42.987501 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f366feb_99b7_4023_bc5d_72a66f8eaefe.slice/crio-7be6347aeab0b9c895e145c0ff517b1777ab12e1f672faf6e54bc79b96a7d4a9 WatchSource:0}: Error finding container 7be6347aeab0b9c895e145c0ff517b1777ab12e1f672faf6e54bc79b96a7d4a9: Status 404 returned error can't find the container with id 7be6347aeab0b9c895e145c0ff517b1777ab12e1f672faf6e54bc79b96a7d4a9 Oct 30 15:54:42.989384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.989357 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d"] Oct 30 15:54:42.990668 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.990634 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:54:42.990754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.990706 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:54:42.990754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:42.990738 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:54:43.022624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:43.022602 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:43.281601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:43.281513 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" event={"ID":"0f366feb-99b7-4023-bc5d-72a66f8eaefe","Type":"ContainerStarted","Data":"e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534"} Oct 30 15:54:43.281601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:43.281564 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" event={"ID":"0f366feb-99b7-4023-bc5d-72a66f8eaefe","Type":"ContainerStarted","Data":"7be6347aeab0b9c895e145c0ff517b1777ab12e1f672faf6e54bc79b96a7d4a9"} Oct 30 15:54:43.337684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:43.337654 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw"] Oct 30 15:54:43.338716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:43.338664 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podStartSLOduration=1.338647295 podStartE2EDuration="1.338647295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:43.333267068 +0000 UTC m=+992.032540232" watchObservedRunningTime="2026-10-30 15:54:43.338647295 +0000 UTC m=+992.037920454" Oct 30 15:54:43.339079 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:54:43.339055 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238833d8_5528_4d55_8e4a_df5ee798ff55.slice/crio-7655f4ca235faf6dd90dc6e9c79579d3ecee45918de456218216493a123c9381 WatchSource:0}: Error finding container 7655f4ca235faf6dd90dc6e9c79579d3ecee45918de456218216493a123c9381: Status 404 returned error can't find the container with id 7655f4ca235faf6dd90dc6e9c79579d3ecee45918de456218216493a123c9381 Oct 30 15:54:43.635236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:43.635208 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:44.292135 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:44.292100 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" event={"ID":"238833d8-5528-4d55-8e4a-df5ee798ff55","Type":"ContainerStarted","Data":"37783ce8b5fea084c85201084ac0edc1aa1b61bd58a7160195a36ab764d98b25"} Oct 30 15:54:44.292314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:44.292150 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" event={"ID":"238833d8-5528-4d55-8e4a-df5ee798ff55","Type":"ContainerStarted","Data":"7655f4ca235faf6dd90dc6e9c79579d3ecee45918de456218216493a123c9381"} Oct 30 15:54:44.340110 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:44.340062 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" podStartSLOduration=2.340049004 podStartE2EDuration="2.340049004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:44.338590236 +0000 UTC m=+993.037863393" watchObservedRunningTime="2026-10-30 15:54:44.340049004 +0000 UTC m=+993.039322231" Oct 30 15:54:44.637291 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:44.637259 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 15:54:44.637468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:44.637308 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 15:54:45.302145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:45.302111 4401 generic.go:356] "Generic (PLEG): container finished" podID="238833d8-5528-4d55-8e4a-df5ee798ff55" containerID="37783ce8b5fea084c85201084ac0edc1aa1b61bd58a7160195a36ab764d98b25" exitCode=0 Oct 30 15:54:45.302357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:45.302333 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" event={"ID":"238833d8-5528-4d55-8e4a-df5ee798ff55","Type":"ContainerDied","Data":"37783ce8b5fea084c85201084ac0edc1aa1b61bd58a7160195a36ab764d98b25"} Oct 30 15:54:45.636377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:45.636336 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 15:54:45.636762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:45.636410 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 15:54:46.509986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.509959 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:46.630939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.630909 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-xks-ca\") pod \"238833d8-5528-4d55-8e4a-df5ee798ff55\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " Oct 30 15:54:46.631141 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.631018 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-openshift-ca\") pod \"238833d8-5528-4d55-8e4a-df5ee798ff55\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " Oct 30 15:54:46.631213 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.631139 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/238833d8-5528-4d55-8e4a-df5ee798ff55-kube-api-access-tw2nc\" (UniqueName: \"kubernetes.io/projected/238833d8-5528-4d55-8e4a-df5ee798ff55-kube-api-access-tw2nc\") pod \"238833d8-5528-4d55-8e4a-df5ee798ff55\" (UID: \"238833d8-5528-4d55-8e4a-df5ee798ff55\") " Oct 30 15:54:46.631535 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.631508 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-openshift-ca" pod "238833d8-5528-4d55-8e4a-df5ee798ff55" (UID: "238833d8-5528-4d55-8e4a-df5ee798ff55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:54:46.631728 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.631708 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:46.633426 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.633392 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-xks-ca" pod "238833d8-5528-4d55-8e4a-df5ee798ff55" (UID: "238833d8-5528-4d55-8e4a-df5ee798ff55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:54:46.633624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.633593 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/238833d8-5528-4d55-8e4a-df5ee798ff55-kube-api-access-tw2nc" pod "238833d8-5528-4d55-8e4a-df5ee798ff55" (UID: "238833d8-5528-4d55-8e4a-df5ee798ff55"). InnerVolumeSpecName "kube-api-access-tw2nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:54:46.635845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.635776 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 15:54:46.635952 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.635839 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 15:54:46.733642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.733605 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw2nc\" (UniqueName: \"kubernetes.io/projected/238833d8-5528-4d55-8e4a-df5ee798ff55-kube-api-access-tw2nc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:46.733642 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:46.733640 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/238833d8-5528-4d55-8e4a-df5ee798ff55-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:47.321490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:47.321459 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" event={"ID":"238833d8-5528-4d55-8e4a-df5ee798ff55","Type":"ContainerDied","Data":"7655f4ca235faf6dd90dc6e9c79579d3ecee45918de456218216493a123c9381"} Oct 30 15:54:47.321696 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:47.321541 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7655f4ca235faf6dd90dc6e9c79579d3ecee45918de456218216493a123c9381" Oct 30 15:54:47.321696 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:47.321606 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw" Oct 30 15:54:47.636046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:47.636009 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 15:54:47.636260 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:47.636069 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 15:54:48.635262 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:48.635223 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 15:54:48.635893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:48.635852 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 15:54:49.635624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:49.635586 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 15:54:49.636162 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:49.635650 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 15:54:50.640422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:50.640387 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:50.640872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:50.640768 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:50.641410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:50.641391 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:54:51.983093 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:54:51.983047 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:54:53.316166 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:53.316121 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:54:53.408018 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:53.407959 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:54:53.423702 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:53.422926 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:54:53.647865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:53.642796 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:54:53.647865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:53.644322 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:54:56.302811 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.302776 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2"] Oct 30 15:54:56.303463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.303449 4401 cpu_manager.go:414] "containerMap: removing container" podUID="238833d8-5528-4d55-8e4a-df5ee798ff55" containerName="revoke-keys" Oct 30 15:54:56.303509 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.303464 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="238833d8-5528-4d55-8e4a-df5ee798ff55" containerName="revoke-keys" Oct 30 15:54:56.303593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.303582 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="238833d8-5528-4d55-8e4a-df5ee798ff55" containerName="revoke-keys" Oct 30 15:54:56.307773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.307754 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.324559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.324530 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:54:56.325215 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.325193 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2"] Oct 30 15:54:56.334865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.333537 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:54:56.334865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.333566 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:54:56.334865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.333669 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:54:56.440059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.440027 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.440230 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.440082 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.440297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.440283 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vh9b\" (UniqueName: \"kubernetes.io/projected/a6aa618c-d769-4e40-bd45-4ef47de32ae7-kube-api-access-7vh9b\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.541410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.541378 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.541565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.541429 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.541638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.541627 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vh9b\" (UniqueName: \"kubernetes.io/projected/a6aa618c-d769-4e40-bd45-4ef47de32ae7-kube-api-access-7vh9b\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.542145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.542122 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.543727 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.543709 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.570444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.570365 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vh9b\" (UniqueName: \"kubernetes.io/projected/a6aa618c-d769-4e40-bd45-4ef47de32ae7-kube-api-access-7vh9b\") pod \"maas-api-revoke-sub-820c1119-d6sn2\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.625548 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.625514 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:56.870097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.870061 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749"] Oct 30 15:54:56.875903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.875878 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:56.911791 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.911758 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749"] Oct 30 15:54:56.947272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.947229 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/5e58ae06-a500-48ab-9882-e1185ab344e2-kube-api-access-dzt4m\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:56.947438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.947282 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:56.947511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.947482 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:56.964231 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:54:56.964196 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa618c_d769_4e40_bd45_4ef47de32ae7.slice/crio-6d91ffcd9f3dcb2f1da678172ec544654a0abbda5edf0aa1f0d7c4d4a7fc4f7f WatchSource:0}: Error finding container 6d91ffcd9f3dcb2f1da678172ec544654a0abbda5edf0aa1f0d7c4d4a7fc4f7f: Status 404 returned error can't find the container with id 6d91ffcd9f3dcb2f1da678172ec544654a0abbda5edf0aa1f0d7c4d4a7fc4f7f Oct 30 15:54:56.966914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:56.966805 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2"] Oct 30 15:54:57.056984 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.056944 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/5e58ae06-a500-48ab-9882-e1185ab344e2-kube-api-access-dzt4m\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:57.057633 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.057607 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:57.066157 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.066124 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:57.072958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.067405 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:57.076615 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.076561 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:57.085530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.085463 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/5e58ae06-a500-48ab-9882-e1185ab344e2-kube-api-access-dzt4m\") pod \"maas-api-revoke-sub-c749da43-z9749\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:57.199606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.199569 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:57.512172 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:54:57.512144 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e58ae06_a500_48ab_9882_e1185ab344e2.slice/crio-6ed983c01d91d6b71988159bc69e95ee5ba58d98bb5368904b2bdf62f985a29e WatchSource:0}: Error finding container 6ed983c01d91d6b71988159bc69e95ee5ba58d98bb5368904b2bdf62f985a29e: Status 404 returned error can't find the container with id 6ed983c01d91d6b71988159bc69e95ee5ba58d98bb5368904b2bdf62f985a29e Oct 30 15:54:57.513336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.513312 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749"] Oct 30 15:54:57.523187 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.523158 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" event={"ID":"5e58ae06-a500-48ab-9882-e1185ab344e2","Type":"ContainerStarted","Data":"6ed983c01d91d6b71988159bc69e95ee5ba58d98bb5368904b2bdf62f985a29e"} Oct 30 15:54:57.524916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.524886 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" event={"ID":"a6aa618c-d769-4e40-bd45-4ef47de32ae7","Type":"ContainerStarted","Data":"fd4110e4ebe6263fdeff7f3d19c7e96c87ccd0c2108cfc5ee72a9314cff4fb93"} Oct 30 15:54:57.524998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.524937 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" event={"ID":"a6aa618c-d769-4e40-bd45-4ef47de32ae7","Type":"ContainerStarted","Data":"6d91ffcd9f3dcb2f1da678172ec544654a0abbda5edf0aa1f0d7c4d4a7fc4f7f"} Oct 30 15:54:57.570658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:57.570602 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" podStartSLOduration=1.570583798 podStartE2EDuration="1.570583798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:57.568501413 +0000 UTC m=+1006.267774570" watchObservedRunningTime="2026-10-30 15:54:57.570583798 +0000 UTC m=+1006.269856955" Oct 30 15:54:58.533567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:58.533525 4401 generic.go:356] "Generic (PLEG): container finished" podID="5e58ae06-a500-48ab-9882-e1185ab344e2" containerID="5f0e24f25bb4f3e785232045d1736da47bfbb630dde4f8d936510edd48ad1e47" exitCode=0 Oct 30 15:54:58.534005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:58.533563 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" event={"ID":"5e58ae06-a500-48ab-9882-e1185ab344e2","Type":"ContainerDied","Data":"5f0e24f25bb4f3e785232045d1736da47bfbb630dde4f8d936510edd48ad1e47"} Oct 30 15:54:58.535238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:58.535216 4401 generic.go:356] "Generic (PLEG): container finished" podID="a6aa618c-d769-4e40-bd45-4ef47de32ae7" containerID="fd4110e4ebe6263fdeff7f3d19c7e96c87ccd0c2108cfc5ee72a9314cff4fb93" exitCode=0 Oct 30 15:54:58.535353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:58.535247 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" event={"ID":"a6aa618c-d769-4e40-bd45-4ef47de32ae7","Type":"ContainerDied","Data":"fd4110e4ebe6263fdeff7f3d19c7e96c87ccd0c2108cfc5ee72a9314cff4fb93"} Oct 30 15:54:59.739327 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.739304 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:54:59.742619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.742601 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:54:59.807041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.806958 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-xks-ca\") pod \"5e58ae06-a500-48ab-9882-e1185ab344e2\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " Oct 30 15:54:59.807041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.807031 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-openshift-ca\") pod \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " Oct 30 15:54:59.807268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.807146 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-xks-ca\") pod \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " Oct 30 15:54:59.807268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.807214 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e58ae06-a500-48ab-9882-e1185ab344e2-kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/5e58ae06-a500-48ab-9882-e1185ab344e2-kube-api-access-dzt4m\") pod \"5e58ae06-a500-48ab-9882-e1185ab344e2\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " Oct 30 15:54:59.807364 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.807326 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-openshift-ca\") pod \"5e58ae06-a500-48ab-9882-e1185ab344e2\" (UID: \"5e58ae06-a500-48ab-9882-e1185ab344e2\") " Oct 30 15:54:59.807420 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.807409 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aa618c-d769-4e40-bd45-4ef47de32ae7-kube-api-access-7vh9b\" (UniqueName: \"kubernetes.io/projected/a6aa618c-d769-4e40-bd45-4ef47de32ae7-kube-api-access-7vh9b\") pod \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\" (UID: \"a6aa618c-d769-4e40-bd45-4ef47de32ae7\") " Oct 30 15:54:59.809563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.809385 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-openshift-ca" pod "5e58ae06-a500-48ab-9882-e1185ab344e2" (UID: "5e58ae06-a500-48ab-9882-e1185ab344e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:54:59.809563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.809408 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-openshift-ca" pod "a6aa618c-d769-4e40-bd45-4ef47de32ae7" (UID: "a6aa618c-d769-4e40-bd45-4ef47de32ae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:54:59.811512 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.811473 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-xks-ca" pod "a6aa618c-d769-4e40-bd45-4ef47de32ae7" (UID: "a6aa618c-d769-4e40-bd45-4ef47de32ae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:54:59.812020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.811919 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa618c-d769-4e40-bd45-4ef47de32ae7-kube-api-access-7vh9b" pod "a6aa618c-d769-4e40-bd45-4ef47de32ae7" (UID: "a6aa618c-d769-4e40-bd45-4ef47de32ae7"). InnerVolumeSpecName "kube-api-access-7vh9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:54:59.812132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.812089 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e58ae06-a500-48ab-9882-e1185ab344e2-kube-api-access-dzt4m" pod "5e58ae06-a500-48ab-9882-e1185ab344e2" (UID: "5e58ae06-a500-48ab-9882-e1185ab344e2"). InnerVolumeSpecName "kube-api-access-dzt4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:54:59.812440 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.812412 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-xks-ca" pod "5e58ae06-a500-48ab-9882-e1185ab344e2" (UID: "5e58ae06-a500-48ab-9882-e1185ab344e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:54:59.909346 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.909310 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:59.909346 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.909340 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:59.909346 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.909349 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6aa618c-d769-4e40-bd45-4ef47de32ae7-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:59.909586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.909358 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/5e58ae06-a500-48ab-9882-e1185ab344e2-kube-api-access-dzt4m\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:59.909586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.909367 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e58ae06-a500-48ab-9882-e1185ab344e2-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:54:59.909586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:54:59.909376 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vh9b\" (UniqueName: \"kubernetes.io/projected/a6aa618c-d769-4e40-bd45-4ef47de32ae7-kube-api-access-7vh9b\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:00.551353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:00.551311 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" event={"ID":"5e58ae06-a500-48ab-9882-e1185ab344e2","Type":"ContainerDied","Data":"6ed983c01d91d6b71988159bc69e95ee5ba58d98bb5368904b2bdf62f985a29e"} Oct 30 15:55:00.551540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:00.551377 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed983c01d91d6b71988159bc69e95ee5ba58d98bb5368904b2bdf62f985a29e" Oct 30 15:55:00.551540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:00.551326 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749" Oct 30 15:55:00.552838 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:00.552799 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" Oct 30 15:55:00.552940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:00.552858 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2" event={"ID":"a6aa618c-d769-4e40-bd45-4ef47de32ae7","Type":"ContainerDied","Data":"6d91ffcd9f3dcb2f1da678172ec544654a0abbda5edf0aa1f0d7c4d4a7fc4f7f"} Oct 30 15:55:00.552940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:00.552899 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d91ffcd9f3dcb2f1da678172ec544654a0abbda5edf0aa1f0d7c4d4a7fc4f7f" Oct 30 15:55:07.811733 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:07.811683 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:55:10.663862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.649710 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:55:10.784466 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.784415 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x"] Oct 30 15:55:10.810182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.810145 4401 cpu_manager.go:414] "containerMap: removing container" podUID="5e58ae06-a500-48ab-9882-e1185ab344e2" containerName="revoke-keys" Oct 30 15:55:10.811273 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.811239 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e58ae06-a500-48ab-9882-e1185ab344e2" containerName="revoke-keys" Oct 30 15:55:10.811495 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.811479 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a6aa618c-d769-4e40-bd45-4ef47de32ae7" containerName="revoke-keys" Oct 30 15:55:10.811594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.811582 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6aa618c-d769-4e40-bd45-4ef47de32ae7" containerName="revoke-keys" Oct 30 15:55:10.819078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.819040 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e58ae06-a500-48ab-9882-e1185ab344e2" containerName="revoke-keys" Oct 30 15:55:10.819330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.819313 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6aa618c-d769-4e40-bd45-4ef47de32ae7" containerName="revoke-keys" Oct 30 15:55:10.841851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.825342 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:10.843326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.843231 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:55:10.883354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.868351 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:55:10.883354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.868418 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:55:10.883354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.868469 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:55:10.915914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.903039 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:10.994857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.994191 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw5mr\" (UniqueName: \"kubernetes.io/projected/e8ea5671-2313-4732-9017-35d41dd17635-kube-api-access-zw5mr\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:10.994857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.994279 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:10.994857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:10.994392 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.046490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.046437 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x"] Oct 30 15:55:11.091415 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.091340 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:11.115264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.107309 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.115264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.107636 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw5mr\" (UniqueName: \"kubernetes.io/projected/e8ea5671-2313-4732-9017-35d41dd17635-kube-api-access-zw5mr\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.115264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.107692 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.115264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.108583 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.202743 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.202656 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw5mr\" (UniqueName: \"kubernetes.io/projected/e8ea5671-2313-4732-9017-35d41dd17635-kube-api-access-zw5mr\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.291853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.289345 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4080bf2-2864x\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.317623 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.317585 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:11.320965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.320921 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:11.382679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.382070 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:55:11.410599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.410041 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:11.639808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:11.639761 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:12.123815 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.123776 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x"] Oct 30 15:55:12.503519 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.503467 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28"] Oct 30 15:55:12.509472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.509447 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.531636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.531603 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28"] Oct 30 15:55:12.591241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591198 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591248 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591349 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591396 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rv2\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-kube-api-access-p6rv2\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591489 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3faa20-1ade-4a35-812b-997c25755b10-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591534 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3faa20-1ade-4a35-812b-997c25755b10-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591700 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591778 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591729 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.591778 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.591765 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693305 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693435 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693488 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693514 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rv2\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-kube-api-access-p6rv2\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693585 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3faa20-1ade-4a35-812b-997c25755b10-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693844 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693636 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3faa20-1ade-4a35-812b-997c25755b10-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693844 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693762 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.693844 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693791 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.694098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693844 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.694098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.693907 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.694098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.694034 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.694296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.694270 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.694595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.694564 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.694696 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.694573 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3faa20-1ade-4a35-812b-997c25755b10-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.697280 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.697030 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3faa20-1ade-4a35-812b-997c25755b10-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.701380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.701345 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.719814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.719779 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rv2\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-kube-api-access-p6rv2\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.719814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.719795 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.831904 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.831874 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:12.836345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.836107 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" event={"ID":"e8ea5671-2313-4732-9017-35d41dd17635","Type":"ContainerStarted","Data":"5699e061caff465211ed2d3269b3ba82f3ddf9bea838e750e38e11c73fad2723"} Oct 30 15:55:12.836345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.836161 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" event={"ID":"e8ea5671-2313-4732-9017-35d41dd17635","Type":"ContainerStarted","Data":"aaf55ea3a0e81ccebfe731b3fc64842ae232f71e075e58436a2981c1f52b3f6a"} Oct 30 15:55:12.882802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:12.882742 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" podStartSLOduration=2.882726529 podStartE2EDuration="2.882726529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:12.881055084 +0000 UTC m=+1021.580328241" watchObservedRunningTime="2026-10-30 15:55:12.882726529 +0000 UTC m=+1021.581999685" Oct 30 15:55:13.163741 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:55:13.163706 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3faa20_1ade_4a35_812b_997c25755b10.slice/crio-f795444981348f60a0788721a97602512f80e0d706b9b3893f1873f540b2d055 WatchSource:0}: Error finding container f795444981348f60a0788721a97602512f80e0d706b9b3893f1873f540b2d055: Status 404 returned error can't find the container with id f795444981348f60a0788721a97602512f80e0d706b9b3893f1873f540b2d055 Oct 30 15:55:13.164974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.164949 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28"] Oct 30 15:55:13.166456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.166419 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:55:13.166554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.166484 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:55:13.166554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.166525 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:55:13.845449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.845410 4401 generic.go:356] "Generic (PLEG): container finished" podID="e8ea5671-2313-4732-9017-35d41dd17635" containerID="5699e061caff465211ed2d3269b3ba82f3ddf9bea838e750e38e11c73fad2723" exitCode=0 Oct 30 15:55:13.845630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.845450 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" event={"ID":"e8ea5671-2313-4732-9017-35d41dd17635","Type":"ContainerDied","Data":"5699e061caff465211ed2d3269b3ba82f3ddf9bea838e750e38e11c73fad2723"} Oct 30 15:55:13.849848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.848286 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" event={"ID":"7e3faa20-1ade-4a35-812b-997c25755b10","Type":"ContainerStarted","Data":"2093ecfc260ab011160bfae9b4642fe2788969f47c7319977f85ba0ee2d8d5bd"} Oct 30 15:55:13.849848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:13.848373 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" event={"ID":"7e3faa20-1ade-4a35-812b-997c25755b10","Type":"ContainerStarted","Data":"f795444981348f60a0788721a97602512f80e0d706b9b3893f1873f540b2d055"} Oct 30 15:55:14.832161 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:14.832120 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:14.834434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:14.834388 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 15:55:14.834583 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:14.834460 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 15:55:14.862853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:14.862366 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podStartSLOduration=2.862343913 podStartE2EDuration="2.862343913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:13.968140677 +0000 UTC m=+1022.667413836" watchObservedRunningTime="2026-10-30 15:55:14.862343913 +0000 UTC m=+1023.561617071" Oct 30 15:55:14.876872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:14.874804 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc"] Oct 30 15:55:14.881880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:14.881818 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:14.888154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:14.888120 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc"] Oct 30 15:55:15.037273 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.037238 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.037490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.037474 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65x9\" (UniqueName: \"kubernetes.io/projected/8d17ed20-b1c9-419b-b8b4-989fc79fc597-kube-api-access-w65x9\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.037536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.037511 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.112521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.112492 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:15.139338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.139306 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65x9\" (UniqueName: \"kubernetes.io/projected/8d17ed20-b1c9-419b-b8b4-989fc79fc597-kube-api-access-w65x9\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.139524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.139352 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.139524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.139409 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.140086 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.140064 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.141928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.141906 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-xks-ca\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.179858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.177557 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65x9\" (UniqueName: \"kubernetes.io/projected/8d17ed20-b1c9-419b-b8b4-989fc79fc597-kube-api-access-w65x9\") pod \"maas-api-revoke-sub-107f9199-4wmxc\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.202604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.202570 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:15.241038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.241007 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ea5671-2313-4732-9017-35d41dd17635-kube-api-access-zw5mr\" (UniqueName: \"kubernetes.io/projected/e8ea5671-2313-4732-9017-35d41dd17635-kube-api-access-zw5mr\") pod \"e8ea5671-2313-4732-9017-35d41dd17635\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " Oct 30 15:55:15.241389 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.241374 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-openshift-ca\") pod \"e8ea5671-2313-4732-9017-35d41dd17635\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " Oct 30 15:55:15.241535 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.241517 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-xks-ca\") pod \"e8ea5671-2313-4732-9017-35d41dd17635\" (UID: \"e8ea5671-2313-4732-9017-35d41dd17635\") " Oct 30 15:55:15.241689 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.241661 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-openshift-ca" pod "e8ea5671-2313-4732-9017-35d41dd17635" (UID: "e8ea5671-2313-4732-9017-35d41dd17635"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:15.242061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.242043 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:15.243447 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.243427 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-xks-ca" pod "e8ea5671-2313-4732-9017-35d41dd17635" (UID: "e8ea5671-2313-4732-9017-35d41dd17635"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:15.243768 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.243724 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ea5671-2313-4732-9017-35d41dd17635-kube-api-access-zw5mr" pod "e8ea5671-2313-4732-9017-35d41dd17635" (UID: "e8ea5671-2313-4732-9017-35d41dd17635"). InnerVolumeSpecName "kube-api-access-zw5mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:15.345817 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.345781 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8ea5671-2313-4732-9017-35d41dd17635-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:15.345817 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.345816 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw5mr\" (UniqueName: \"kubernetes.io/projected/e8ea5671-2313-4732-9017-35d41dd17635-kube-api-access-zw5mr\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:15.561868 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:55:15.545888 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d17ed20_b1c9_419b_b8b4_989fc79fc597.slice/crio-dc66da3b2291d45ab663f6e8a89cc215e74b4dcd986b1d52b6b12b0fcb461739 WatchSource:0}: Error finding container dc66da3b2291d45ab663f6e8a89cc215e74b4dcd986b1d52b6b12b0fcb461739: Status 404 returned error can't find the container with id dc66da3b2291d45ab663f6e8a89cc215e74b4dcd986b1d52b6b12b0fcb461739 Oct 30 15:55:15.571864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.568428 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc"] Oct 30 15:55:15.832764 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.832728 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 15:55:15.833291 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.832791 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 15:55:15.890603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.890519 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" Oct 30 15:55:15.890603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.890548 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x" event={"ID":"e8ea5671-2313-4732-9017-35d41dd17635","Type":"ContainerDied","Data":"aaf55ea3a0e81ccebfe731b3fc64842ae232f71e075e58436a2981c1f52b3f6a"} Oct 30 15:55:15.890603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.890595 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf55ea3a0e81ccebfe731b3fc64842ae232f71e075e58436a2981c1f52b3f6a" Oct 30 15:55:15.892164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:15.892137 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" event={"ID":"8d17ed20-b1c9-419b-b8b4-989fc79fc597","Type":"ContainerStarted","Data":"dc66da3b2291d45ab663f6e8a89cc215e74b4dcd986b1d52b6b12b0fcb461739"} Oct 30 15:55:16.833003 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:16.832968 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 15:55:16.833417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:16.833021 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 15:55:16.901585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:16.901549 4401 generic.go:356] "Generic (PLEG): container finished" podID="8d17ed20-b1c9-419b-b8b4-989fc79fc597" containerID="7c04c702091d22729d7b27bd068437012922d66798e86e1772af3649ba5df546" exitCode=0 Oct 30 15:55:16.901759 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:16.901601 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" event={"ID":"8d17ed20-b1c9-419b-b8b4-989fc79fc597","Type":"ContainerDied","Data":"7c04c702091d22729d7b27bd068437012922d66798e86e1772af3649ba5df546"} Oct 30 15:55:17.729809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.729769 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2"] Oct 30 15:55:17.730491 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.730476 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e8ea5671-2313-4732-9017-35d41dd17635" containerName="revoke-keys" Oct 30 15:55:17.730547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.730492 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ea5671-2313-4732-9017-35d41dd17635" containerName="revoke-keys" Oct 30 15:55:17.730623 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.730612 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ea5671-2313-4732-9017-35d41dd17635" containerName="revoke-keys" Oct 30 15:55:17.734484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.734461 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.777994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.770080 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2"] Oct 30 15:55:17.824697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.824661 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctsf\" (UniqueName: \"kubernetes.io/projected/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-kube-api-access-vctsf\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.824928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.824756 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.824928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.824786 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.832657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.832628 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 15:55:17.832768 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.832694 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 15:55:17.930848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.926451 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctsf\" (UniqueName: \"kubernetes.io/projected/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-kube-api-access-vctsf\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.930848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.926629 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.930848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.926671 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.947852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.939841 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:17.981739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:17.978431 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:18.019755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.018532 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctsf\" (UniqueName: \"kubernetes.io/projected/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-kube-api-access-vctsf\") pod \"maas-api-revoke-sub-1de807e0-876l2\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:18.077151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.077111 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:18.280252 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.280172 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:18.439056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.438811 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-xks-ca\") pod \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " Oct 30 15:55:18.439056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.438898 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d17ed20-b1c9-419b-b8b4-989fc79fc597-kube-api-access-w65x9\" (UniqueName: \"kubernetes.io/projected/8d17ed20-b1c9-419b-b8b4-989fc79fc597-kube-api-access-w65x9\") pod \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " Oct 30 15:55:18.439056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.439004 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-openshift-ca\") pod \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\" (UID: \"8d17ed20-b1c9-419b-b8b4-989fc79fc597\") " Oct 30 15:55:18.440067 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.440039 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-openshift-ca" pod "8d17ed20-b1c9-419b-b8b4-989fc79fc597" (UID: "8d17ed20-b1c9-419b-b8b4-989fc79fc597"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:18.441867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.441811 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-xks-ca" pod "8d17ed20-b1c9-419b-b8b4-989fc79fc597" (UID: "8d17ed20-b1c9-419b-b8b4-989fc79fc597"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:18.442236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.442202 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d17ed20-b1c9-419b-b8b4-989fc79fc597-kube-api-access-w65x9" pod "8d17ed20-b1c9-419b-b8b4-989fc79fc597" (UID: "8d17ed20-b1c9-419b-b8b4-989fc79fc597"). InnerVolumeSpecName "kube-api-access-w65x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:18.442993 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:55:18.442966 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7847a62_f8dd_43ce_b5cb_53ba70ba5fae.slice/crio-8dca362d81a56bcd4cf965c013998d783553c99e3d24e62684b4ff56a22b11f5 WatchSource:0}: Error finding container 8dca362d81a56bcd4cf965c013998d783553c99e3d24e62684b4ff56a22b11f5: Status 404 returned error can't find the container with id 8dca362d81a56bcd4cf965c013998d783553c99e3d24e62684b4ff56a22b11f5 Oct 30 15:55:18.443628 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.443554 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2"] Oct 30 15:55:18.541895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.541798 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:18.541895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.541846 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w65x9\" (UniqueName: \"kubernetes.io/projected/8d17ed20-b1c9-419b-b8b4-989fc79fc597-kube-api-access-w65x9\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:18.541895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.541863 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d17ed20-b1c9-419b-b8b4-989fc79fc597-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:18.833277 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.833191 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 15:55:18.833277 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.833260 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 15:55:18.990665 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.990616 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" event={"ID":"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae","Type":"ContainerStarted","Data":"05f98370ea814e3227e916f8470c32a7f7ae813af254860ac11089cf2e374da2"} Oct 30 15:55:18.991338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.991314 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" event={"ID":"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae","Type":"ContainerStarted","Data":"8dca362d81a56bcd4cf965c013998d783553c99e3d24e62684b4ff56a22b11f5"} Oct 30 15:55:18.992385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.992364 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" Oct 30 15:55:18.992505 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.992488 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc" event={"ID":"8d17ed20-b1c9-419b-b8b4-989fc79fc597","Type":"ContainerDied","Data":"dc66da3b2291d45ab663f6e8a89cc215e74b4dcd986b1d52b6b12b0fcb461739"} Oct 30 15:55:18.992580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:18.992570 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc66da3b2291d45ab663f6e8a89cc215e74b4dcd986b1d52b6b12b0fcb461739" Oct 30 15:55:19.036249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:19.036193 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" podStartSLOduration=2.036178257 podStartE2EDuration="2.036178257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:19.03472823 +0000 UTC m=+1027.734001411" watchObservedRunningTime="2026-10-30 15:55:19.036178257 +0000 UTC m=+1027.735451413" Oct 30 15:55:19.836130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:19.833256 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 15:55:19.836130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:19.833320 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 15:55:20.004371 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:20.004333 4401 generic.go:356] "Generic (PLEG): container finished" podID="b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" containerID="05f98370ea814e3227e916f8470c32a7f7ae813af254860ac11089cf2e374da2" exitCode=0 Oct 30 15:55:20.004883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:20.004422 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" event={"ID":"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae","Type":"ContainerDied","Data":"05f98370ea814e3227e916f8470c32a7f7ae813af254860ac11089cf2e374da2"} Oct 30 15:55:20.838004 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:20.837970 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:20.838485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:20.838465 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:20.839481 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:20.839455 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:55:21.204966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.204932 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:21.306114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.306081 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-openshift-ca\") pod \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " Oct 30 15:55:21.306295 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.306251 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-kube-api-access-vctsf\" (UniqueName: \"kubernetes.io/projected/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-kube-api-access-vctsf\") pod \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " Oct 30 15:55:21.306347 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.306303 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-xks-ca\") pod \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\" (UID: \"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae\") " Oct 30 15:55:21.306593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.306560 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-openshift-ca" pod "b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" (UID: "b7847a62-f8dd-43ce-b5cb-53ba70ba5fae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:21.306887 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.306865 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:21.308774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.308747 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-kube-api-access-vctsf" pod "b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" (UID: "b7847a62-f8dd-43ce-b5cb-53ba70ba5fae"). InnerVolumeSpecName "kube-api-access-vctsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:21.308923 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.308747 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-xks-ca" pod "b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" (UID: "b7847a62-f8dd-43ce-b5cb-53ba70ba5fae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:21.408055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.408017 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctsf\" (UniqueName: \"kubernetes.io/projected/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-kube-api-access-vctsf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:21.408055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:21.408055 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:21.969417 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:55:21.969378 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:55:22.024038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:22.024007 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" Oct 30 15:55:22.024223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:22.024010 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2" event={"ID":"b7847a62-f8dd-43ce-b5cb-53ba70ba5fae","Type":"ContainerDied","Data":"8dca362d81a56bcd4cf965c013998d783553c99e3d24e62684b4ff56a22b11f5"} Oct 30 15:55:22.024223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:22.024135 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dca362d81a56bcd4cf965c013998d783553c99e3d24e62684b4ff56a22b11f5" Oct 30 15:55:26.674794 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:26.673938 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:26.877343 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:26.877299 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:27.829847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.829787 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8"] Oct 30 15:55:27.831306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.830868 4401 cpu_manager.go:414] "containerMap: removing container" podUID="8d17ed20-b1c9-419b-b8b4-989fc79fc597" containerName="revoke-keys" Oct 30 15:55:27.831306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.830893 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d17ed20-b1c9-419b-b8b4-989fc79fc597" containerName="revoke-keys" Oct 30 15:55:27.831306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.830937 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" containerName="revoke-keys" Oct 30 15:55:27.831306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.830945 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" containerName="revoke-keys" Oct 30 15:55:27.831306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.831137 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d17ed20-b1c9-419b-b8b4-989fc79fc597" containerName="revoke-keys" Oct 30 15:55:27.831306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.831155 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" containerName="revoke-keys" Oct 30 15:55:27.836935 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.836903 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8"] Oct 30 15:55:27.837100 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.837026 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:27.858861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.858797 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:55:27.859801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.859775 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:55:27.860089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.860072 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:55:27.860358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.860323 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:55:27.891707 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.891666 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:27.892110 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.892087 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqh9r\" (UniqueName: \"kubernetes.io/projected/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-kube-api-access-sqh9r\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:27.892442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.892425 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:27.995622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.995549 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:27.995904 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.995695 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:27.999910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:27.999866 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqh9r\" (UniqueName: \"kubernetes.io/projected/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-kube-api-access-sqh9r\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:28.002353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:28.002315 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:28.016210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:28.016166 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:28.025709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:28.025676 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqh9r\" (UniqueName: \"kubernetes.io/projected/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-kube-api-access-sqh9r\") pod \"maas-api-revoke-sub-3c3e6ae0-lmmj8\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:28.197974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:28.197943 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:28.523506 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:28.523463 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8"] Oct 30 15:55:29.200151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:29.200103 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" event={"ID":"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65","Type":"ContainerStarted","Data":"473f5bb960c77d85d040140f35dcca0287daaf0d78e4de5ab8088b74b68bd84b"} Oct 30 15:55:29.200538 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:29.200176 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" event={"ID":"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65","Type":"ContainerStarted","Data":"fc94d9cb07f5461400b3247ffb8dc9958889c5923b333a4a9892c92dfc7649cb"} Oct 30 15:55:29.248338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:29.248272 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" podStartSLOduration=2.248254661 podStartE2EDuration="2.248254661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:29.243851003 +0000 UTC m=+1037.943124151" watchObservedRunningTime="2026-10-30 15:55:29.248254661 +0000 UTC m=+1037.947527818" Oct 30 15:55:30.213762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:30.213723 4401 generic.go:356] "Generic (PLEG): container finished" podID="e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" containerID="473f5bb960c77d85d040140f35dcca0287daaf0d78e4de5ab8088b74b68bd84b" exitCode=0 Oct 30 15:55:30.214234 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:30.213810 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" event={"ID":"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65","Type":"ContainerDied","Data":"473f5bb960c77d85d040140f35dcca0287daaf0d78e4de5ab8088b74b68bd84b"} Oct 30 15:55:31.418524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.418496 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:31.573140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.573036 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-openshift-ca\") pod \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " Oct 30 15:55:31.573323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.573173 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-kube-api-access-sqh9r\" (UniqueName: \"kubernetes.io/projected/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-kube-api-access-sqh9r\") pod \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " Oct 30 15:55:31.573323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.573232 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-xks-ca\") pod \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\" (UID: \"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65\") " Oct 30 15:55:31.573511 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.573495 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-openshift-ca" pod "e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" (UID: "e3c07d02-b5ef-49cf-8c4e-bc693da6ef65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:31.574060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.574031 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:31.575453 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.575424 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-kube-api-access-sqh9r" pod "e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" (UID: "e3c07d02-b5ef-49cf-8c4e-bc693da6ef65"). InnerVolumeSpecName "kube-api-access-sqh9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:31.575547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.575527 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-xks-ca" pod "e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" (UID: "e3c07d02-b5ef-49cf-8c4e-bc693da6ef65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:31.675136 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.675102 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqh9r\" (UniqueName: \"kubernetes.io/projected/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-kube-api-access-sqh9r\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:31.675136 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:31.675131 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:32.243175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:32.243135 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" Oct 30 15:55:32.243364 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:32.243172 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8" event={"ID":"e3c07d02-b5ef-49cf-8c4e-bc693da6ef65","Type":"ContainerDied","Data":"fc94d9cb07f5461400b3247ffb8dc9958889c5923b333a4a9892c92dfc7649cb"} Oct 30 15:55:32.243364 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:32.243225 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc94d9cb07f5461400b3247ffb8dc9958889c5923b333a4a9892c92dfc7649cb" Oct 30 15:55:34.941060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.932188 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7"] Oct 30 15:55:34.941060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.934104 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" containerName="revoke-keys" Oct 30 15:55:34.941060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.934132 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" containerName="revoke-keys" Oct 30 15:55:34.941060 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.934326 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" containerName="revoke-keys" Oct 30 15:55:34.944377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.944346 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:34.958775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.957070 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7"] Oct 30 15:55:34.976059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.976019 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:55:34.976234 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.976068 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:55:34.976234 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.976104 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:55:34.976543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:34.976523 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:55:35.033412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.033368 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z892x\" (UniqueName: \"kubernetes.io/projected/b0a12856-d219-4301-b70c-79c0cbcf4109-kube-api-access-z892x\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.033612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.033588 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.033737 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.033724 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.135048 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.135010 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.135260 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.135076 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z892x\" (UniqueName: \"kubernetes.io/projected/b0a12856-d219-4301-b70c-79c0cbcf4109-kube-api-access-z892x\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.135260 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.135195 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.135747 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.135724 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.137663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.137642 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.160210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.160174 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z892x\" (UniqueName: \"kubernetes.io/projected/b0a12856-d219-4301-b70c-79c0cbcf4109-kube-api-access-z892x\") pod \"maas-api-revoke-sub-5bc695fd-rvkh7\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.278517 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.278429 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:35.592601 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:55:35.592557 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0a12856_d219_4301_b70c_79c0cbcf4109.slice/crio-ec4785f9eacf52b9fcdca24c6b6f5f20804fc440011ce1035cd95367cac1f756 WatchSource:0}: Error finding container ec4785f9eacf52b9fcdca24c6b6f5f20804fc440011ce1035cd95367cac1f756: Status 404 returned error can't find the container with id ec4785f9eacf52b9fcdca24c6b6f5f20804fc440011ce1035cd95367cac1f756 Oct 30 15:55:35.596455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:35.596408 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7"] Oct 30 15:55:36.283294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:36.283207 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" event={"ID":"b0a12856-d219-4301-b70c-79c0cbcf4109","Type":"ContainerStarted","Data":"eb65710cb67441b094c60311d04b21fef89e7edc34543d47256903f012a11231"} Oct 30 15:55:36.283294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:36.283265 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" event={"ID":"b0a12856-d219-4301-b70c-79c0cbcf4109","Type":"ContainerStarted","Data":"ec4785f9eacf52b9fcdca24c6b6f5f20804fc440011ce1035cd95367cac1f756"} Oct 30 15:55:36.329867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:36.329780 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" podStartSLOduration=2.329765193 podStartE2EDuration="2.329765193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:36.327081233 +0000 UTC m=+1045.026354391" watchObservedRunningTime="2026-10-30 15:55:36.329765193 +0000 UTC m=+1045.029038349" Oct 30 15:55:37.239657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.239622 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j"] Oct 30 15:55:37.244312 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.244286 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.264007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.263966 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j"] Oct 30 15:55:37.297647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.297605 4401 generic.go:356] "Generic (PLEG): container finished" podID="b0a12856-d219-4301-b70c-79c0cbcf4109" containerID="eb65710cb67441b094c60311d04b21fef89e7edc34543d47256903f012a11231" exitCode=0 Oct 30 15:55:37.298124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.297695 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" event={"ID":"b0a12856-d219-4301-b70c-79c0cbcf4109","Type":"ContainerDied","Data":"eb65710cb67441b094c60311d04b21fef89e7edc34543d47256903f012a11231"} Oct 30 15:55:37.367226 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.367188 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwq2d\" (UniqueName: \"kubernetes.io/projected/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-kube-api-access-lwq2d\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.367445 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.367421 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.367552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.367489 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.469089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.469050 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.469271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.469226 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.469405 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.469386 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwq2d\" (UniqueName: \"kubernetes.io/projected/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-kube-api-access-lwq2d\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.469878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.469856 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.472147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.472116 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.495007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.494932 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwq2d\" (UniqueName: \"kubernetes.io/projected/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-kube-api-access-lwq2d\") pod \"maas-api-revoke-sub-1669e612-66k4j\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.561572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.561533 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:37.874286 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:55:37.874254 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca77415d_2c8c_4c1f_8e1f_8c5642b10a75.slice/crio-f8e99e22ded724eaa9db5c6b3437bffa40e2c680b970694af5e0c837e1a23158 WatchSource:0}: Error finding container f8e99e22ded724eaa9db5c6b3437bffa40e2c680b970694af5e0c837e1a23158: Status 404 returned error can't find the container with id f8e99e22ded724eaa9db5c6b3437bffa40e2c680b970694af5e0c837e1a23158 Oct 30 15:55:37.874714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:37.874685 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j"] Oct 30 15:55:38.309595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.309550 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" event={"ID":"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75","Type":"ContainerStarted","Data":"3d9e4ae7d8a5e8f27bbaaf4e5f321d50bf7fbba3b728cf5907f011a31096a636"} Oct 30 15:55:38.310068 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.309646 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" event={"ID":"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75","Type":"ContainerStarted","Data":"f8e99e22ded724eaa9db5c6b3437bffa40e2c680b970694af5e0c837e1a23158"} Oct 30 15:55:38.356399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.356341 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" podStartSLOduration=1.3563225939999999 podStartE2EDuration="1.356322594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:38.352523133 +0000 UTC m=+1047.051796291" watchObservedRunningTime="2026-10-30 15:55:38.356322594 +0000 UTC m=+1047.055595798" Oct 30 15:55:38.586375 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.586346 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:38.699979 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.699939 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-xks-ca\") pod \"b0a12856-d219-4301-b70c-79c0cbcf4109\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " Oct 30 15:55:38.700185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.700152 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0a12856-d219-4301-b70c-79c0cbcf4109-kube-api-access-z892x\" (UniqueName: \"kubernetes.io/projected/b0a12856-d219-4301-b70c-79c0cbcf4109-kube-api-access-z892x\") pod \"b0a12856-d219-4301-b70c-79c0cbcf4109\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " Oct 30 15:55:38.700258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.700219 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-openshift-ca\") pod \"b0a12856-d219-4301-b70c-79c0cbcf4109\" (UID: \"b0a12856-d219-4301-b70c-79c0cbcf4109\") " Oct 30 15:55:38.700592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.700563 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-openshift-ca" pod "b0a12856-d219-4301-b70c-79c0cbcf4109" (UID: "b0a12856-d219-4301-b70c-79c0cbcf4109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:38.700800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.700773 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:38.702327 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.702302 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-xks-ca" pod "b0a12856-d219-4301-b70c-79c0cbcf4109" (UID: "b0a12856-d219-4301-b70c-79c0cbcf4109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:38.702444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.702396 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0a12856-d219-4301-b70c-79c0cbcf4109-kube-api-access-z892x" pod "b0a12856-d219-4301-b70c-79c0cbcf4109" (UID: "b0a12856-d219-4301-b70c-79c0cbcf4109"). InnerVolumeSpecName "kube-api-access-z892x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:38.802445 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.802394 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0a12856-d219-4301-b70c-79c0cbcf4109-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:38.802445 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:38.802437 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z892x\" (UniqueName: \"kubernetes.io/projected/b0a12856-d219-4301-b70c-79c0cbcf4109-kube-api-access-z892x\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:39.321798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:39.321705 4401 generic.go:356] "Generic (PLEG): container finished" podID="ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" containerID="3d9e4ae7d8a5e8f27bbaaf4e5f321d50bf7fbba3b728cf5907f011a31096a636" exitCode=0 Oct 30 15:55:39.322332 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:39.321789 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" event={"ID":"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75","Type":"ContainerDied","Data":"3d9e4ae7d8a5e8f27bbaaf4e5f321d50bf7fbba3b728cf5907f011a31096a636"} Oct 30 15:55:39.323352 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:39.323326 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" event={"ID":"b0a12856-d219-4301-b70c-79c0cbcf4109","Type":"ContainerDied","Data":"ec4785f9eacf52b9fcdca24c6b6f5f20804fc440011ce1035cd95367cac1f756"} Oct 30 15:55:39.323453 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:39.323369 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec4785f9eacf52b9fcdca24c6b6f5f20804fc440011ce1035cd95367cac1f756" Oct 30 15:55:39.323453 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:39.323370 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7" Oct 30 15:55:40.511171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.511143 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:40.644913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.644870 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-openshift-ca\") pod \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " Oct 30 15:55:40.645208 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.645180 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-xks-ca\") pod \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " Oct 30 15:55:40.645315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.645282 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-openshift-ca" pod "ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" (UID: "ca77415d-2c8c-4c1f-8e1f-8c5642b10a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:40.645315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.645301 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-kube-api-access-lwq2d\" (UniqueName: \"kubernetes.io/projected/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-kube-api-access-lwq2d\") pod \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\" (UID: \"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75\") " Oct 30 15:55:40.645990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.645964 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:40.647592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.647566 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-xks-ca" pod "ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" (UID: "ca77415d-2c8c-4c1f-8e1f-8c5642b10a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:40.647942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.647917 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-kube-api-access-lwq2d" pod "ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" (UID: "ca77415d-2c8c-4c1f-8e1f-8c5642b10a75"). InnerVolumeSpecName "kube-api-access-lwq2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:40.747407 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.747365 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:40.747407 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:40.747403 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwq2d\" (UniqueName: \"kubernetes.io/projected/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75-kube-api-access-lwq2d\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:41.341340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:41.341307 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" Oct 30 15:55:41.341520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:41.341339 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j" event={"ID":"ca77415d-2c8c-4c1f-8e1f-8c5642b10a75","Type":"ContainerDied","Data":"f8e99e22ded724eaa9db5c6b3437bffa40e2c680b970694af5e0c837e1a23158"} Oct 30 15:55:41.341520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:41.341425 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8e99e22ded724eaa9db5c6b3437bffa40e2c680b970694af5e0c837e1a23158" Oct 30 15:55:41.861910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:41.861876 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:55:42.717849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.716908 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw"] Oct 30 15:55:42.718117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.718062 4401 cpu_manager.go:414] "containerMap: removing container" podUID="ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" containerName="revoke-keys" Oct 30 15:55:42.718117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.718098 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" containerName="revoke-keys" Oct 30 15:55:42.718261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.718151 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b0a12856-d219-4301-b70c-79c0cbcf4109" containerName="revoke-keys" Oct 30 15:55:42.718261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.718164 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0a12856-d219-4301-b70c-79c0cbcf4109" containerName="revoke-keys" Oct 30 15:55:42.718367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.718359 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" containerName="revoke-keys" Oct 30 15:55:42.718439 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.718379 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0a12856-d219-4301-b70c-79c0cbcf4109" containerName="revoke-keys" Oct 30 15:55:42.740872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.731472 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.743368 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.743327 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw"] Oct 30 15:55:42.842078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842038 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842148 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2d8d\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-kube-api-access-n2d8d\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842195 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842291 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842330 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8176a4-c357-45be-b324-ccbb8c53ee36-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842567 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842707 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842687 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842777 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842747 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.842876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.842858 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.945091 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.945046 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.945692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.945643 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.945989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.945970 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8176a4-c357-45be-b324-ccbb8c53ee36-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.946210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.945578 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-data\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.946771 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.946300 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.946873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.946583 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-certs\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.946946 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.946925 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.947039 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.947013 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.947448 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.947425 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.947988 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.947858 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.948258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.948237 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.948762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.948633 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-credential-socket\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.948946 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.948924 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2d8d\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-kube-api-access-n2d8d\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.949441 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.949056 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-podinfo\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.949911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.949883 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-envoy\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.950581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.950557 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8176a4-c357-45be-b324-ccbb8c53ee36-istiod-ca-cert\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.973127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.973033 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-token\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:42.974326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:42.974290 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2d8d\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-kube-api-access-n2d8d\") pod \"e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:43.120850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:43.115542 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:44.340861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.323063 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:44.340861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.326736 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:55:44.390807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.390747 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:44.448579 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.442944 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:55:44.539853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.529402 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": context deadline exceeded" Oct 30 15:55:44.644978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.642201 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:44.841851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.825153 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:55:44.841851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.836174 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:55:44.841851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.836486 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:55:44.841851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.836797 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:55:44.978908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:44.947612 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:55:45.253910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.253379 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw"] Oct 30 15:55:45.429863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.427573 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:45.444883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.436450 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:55:45.447181 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.447142 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:55:45.466345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.466291 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:45.467859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.467801 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:55:45.467993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.467907 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:55:45.490097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.468894 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:55:45.672002 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.671947 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:45.932850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.915961 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:46.001861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:45.974901 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:55:46.253103 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.127491 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:55:46.289373 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.289346 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:55:46.473592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.473536 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:55:46.475087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.449918 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:55:46.493421 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.468927 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:55:46.538248 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.429908 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:55:46.538248 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.475045 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:55:46.538248 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.515180 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:55:46.612050 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.612005 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" event={"ID":"cf8176a4-c357-45be-b324-ccbb8c53ee36","Type":"ContainerStarted","Data":"549e61829518b61242df4c7c495280c95202a0bb5b7b3dd710ec2d445d2ec6fe"} Oct 30 15:55:46.723847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.702251 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:47.005042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:46.923041 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:47.032850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.003199 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:47.118861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.118809 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:55:47.145909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.140160 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:55:47.162079 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.162025 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:55:47.192440 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.191161 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:55:47.216909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.216511 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:55:47.335270 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.335175 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Oct 30 15:55:47.347469 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.347337 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:55:47.364267 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.364236 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:55:47.379509 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.379295 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:55:47.623864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.623779 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" event={"ID":"cf8176a4-c357-45be-b324-ccbb8c53ee36","Type":"ContainerStarted","Data":"c8780163b9f8e75f85b8e2992366edcb935fd7e42a21e1d2778c0d3ce135f092"} Oct 30 15:55:47.674241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:47.674183 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podStartSLOduration=5.674168203 podStartE2EDuration="5.674168203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:47.671960096 +0000 UTC m=+1056.371233276" watchObservedRunningTime="2026-10-30 15:55:47.674168203 +0000 UTC m=+1056.373441360" Oct 30 15:55:48.116809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:48.116766 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:48.118264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:48.118234 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 15:55:48.118416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:48.118289 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 15:55:49.116509 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:49.116475 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 15:55:49.116997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:49.116528 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 15:55:50.117045 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:50.116998 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 15:55:50.117559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:50.117056 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 15:55:51.116933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:51.116891 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 15:55:51.117134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:51.116959 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 15:55:51.991915 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:55:51.991876 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:55:52.041926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.041886 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp"] Oct 30 15:55:52.055924 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.054674 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.079678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.079641 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:55:52.082235 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.082205 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:55:52.082715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.082686 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp"] Oct 30 15:55:52.090635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.090602 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:55:52.090806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.090666 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:55:52.117065 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.117030 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 15:55:52.117216 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.117096 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 15:55:52.179387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.179346 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.179657 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.179636 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.179732 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.179680 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nm4v\" (UniqueName: \"kubernetes.io/projected/af60c524-888d-4f3c-8ea0-40899b674491-kube-api-access-7nm4v\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.280985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.280890 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.280985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.280955 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nm4v\" (UniqueName: \"kubernetes.io/projected/af60c524-888d-4f3c-8ea0-40899b674491-kube-api-access-7nm4v\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.281209 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.280991 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.281692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.281670 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.283385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.283355 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.312104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.312068 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nm4v\" (UniqueName: \"kubernetes.io/projected/af60c524-888d-4f3c-8ea0-40899b674491-kube-api-access-7nm4v\") pod \"maas-api-revoke-sub-77233428-cz9mp\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.407674 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.407631 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:52.766874 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:55:52.746124 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf60c524_888d_4f3c_8ea0_40899b674491.slice/crio-7e267df9aed59f2b5ec3512780981d2b919e57e9606096bcfb873079212bd563 WatchSource:0}: Error finding container 7e267df9aed59f2b5ec3512780981d2b919e57e9606096bcfb873079212bd563: Status 404 returned error can't find the container with id 7e267df9aed59f2b5ec3512780981d2b919e57e9606096bcfb873079212bd563 Oct 30 15:55:52.766874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:52.760684 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp"] Oct 30 15:55:53.142852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:53.139646 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 15:55:53.142852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:53.139745 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 15:55:53.761613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:53.761570 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" event={"ID":"af60c524-888d-4f3c-8ea0-40899b674491","Type":"ContainerStarted","Data":"e4246b2a22dbf7898bab8fcb652a0d2d769f4338349515ebbecd4dcfb1d112fd"} Oct 30 15:55:53.761846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:53.761635 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" event={"ID":"af60c524-888d-4f3c-8ea0-40899b674491","Type":"ContainerStarted","Data":"7e267df9aed59f2b5ec3512780981d2b919e57e9606096bcfb873079212bd563"} Oct 30 15:55:53.808054 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:53.807983 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" podStartSLOduration=1.8079682479999999 podStartE2EDuration="1.807968248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:53.805080638 +0000 UTC m=+1062.504353794" watchObservedRunningTime="2026-10-30 15:55:53.807968248 +0000 UTC m=+1062.507241465" Oct 30 15:55:54.116792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.116749 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 15:55:54.117037 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.116815 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 15:55:54.519522 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.519428 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b"] Oct 30 15:55:54.524466 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.524437 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.547306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.547265 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b"] Oct 30 15:55:54.618756 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.618712 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbkm\" (UniqueName: \"kubernetes.io/projected/9c1b189e-222c-42ee-827c-55653cc72209-kube-api-access-fwbkm\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.618756 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.618757 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.619062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.618911 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.721151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.721108 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.721509 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.721481 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbkm\" (UniqueName: \"kubernetes.io/projected/9c1b189e-222c-42ee-827c-55653cc72209-kube-api-access-fwbkm\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.721595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.721531 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.722438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.722400 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.723562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.723520 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.749695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.749660 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbkm\" (UniqueName: \"kubernetes.io/projected/9c1b189e-222c-42ee-827c-55653cc72209-kube-api-access-fwbkm\") pod \"maas-api-revoke-sub-bfe6cc45-6hf9b\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:54.778748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.778642 4401 generic.go:356] "Generic (PLEG): container finished" podID="af60c524-888d-4f3c-8ea0-40899b674491" containerID="e4246b2a22dbf7898bab8fcb652a0d2d769f4338349515ebbecd4dcfb1d112fd" exitCode=0 Oct 30 15:55:54.779083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.779057 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" event={"ID":"af60c524-888d-4f3c-8ea0-40899b674491","Type":"ContainerDied","Data":"e4246b2a22dbf7898bab8fcb652a0d2d769f4338349515ebbecd4dcfb1d112fd"} Oct 30 15:55:54.848958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:54.848907 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:55.154933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.154152 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:55.160340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.159571 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:55.161209 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.161123 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:55:55.335740 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:55:55.335710 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1b189e_222c_42ee_827c_55653cc72209.slice/crio-d341525409de6f62219d1bac90205e6a91eb40e4fd34e45b2accfbe6419444d4 WatchSource:0}: Error finding container d341525409de6f62219d1bac90205e6a91eb40e4fd34e45b2accfbe6419444d4: Status 404 returned error can't find the container with id d341525409de6f62219d1bac90205e6a91eb40e4fd34e45b2accfbe6419444d4 Oct 30 15:55:55.339012 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.338984 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b"] Oct 30 15:55:55.789067 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.789027 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" event={"ID":"9c1b189e-222c-42ee-827c-55653cc72209","Type":"ContainerStarted","Data":"54401bea0a911e95f8683b661f740bb74cd4371eb30738197b51be72ad628246"} Oct 30 15:55:55.789589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.789087 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" event={"ID":"9c1b189e-222c-42ee-827c-55653cc72209","Type":"ContainerStarted","Data":"d341525409de6f62219d1bac90205e6a91eb40e4fd34e45b2accfbe6419444d4"} Oct 30 15:55:55.835177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.835120 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" podStartSLOduration=1.835103325 podStartE2EDuration="1.835103325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:55:55.832136611 +0000 UTC m=+1064.531409769" watchObservedRunningTime="2026-10-30 15:55:55.835103325 +0000 UTC m=+1064.534376481" Oct 30 15:55:55.955102 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:55.955071 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:56.046424 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.046381 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-openshift-ca\") pod \"af60c524-888d-4f3c-8ea0-40899b674491\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " Oct 30 15:55:56.046599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.046462 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af60c524-888d-4f3c-8ea0-40899b674491-kube-api-access-7nm4v\" (UniqueName: \"kubernetes.io/projected/af60c524-888d-4f3c-8ea0-40899b674491-kube-api-access-7nm4v\") pod \"af60c524-888d-4f3c-8ea0-40899b674491\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " Oct 30 15:55:56.046599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.046544 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-xks-ca\") pod \"af60c524-888d-4f3c-8ea0-40899b674491\" (UID: \"af60c524-888d-4f3c-8ea0-40899b674491\") " Oct 30 15:55:56.046877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.046805 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-openshift-ca" pod "af60c524-888d-4f3c-8ea0-40899b674491" (UID: "af60c524-888d-4f3c-8ea0-40899b674491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:56.047322 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.047292 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:56.048852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.048815 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-xks-ca" pod "af60c524-888d-4f3c-8ea0-40899b674491" (UID: "af60c524-888d-4f3c-8ea0-40899b674491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:56.048968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.048941 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af60c524-888d-4f3c-8ea0-40899b674491-kube-api-access-7nm4v" pod "af60c524-888d-4f3c-8ea0-40899b674491" (UID: "af60c524-888d-4f3c-8ea0-40899b674491"). InnerVolumeSpecName "kube-api-access-7nm4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:56.148789 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.148755 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af60c524-888d-4f3c-8ea0-40899b674491-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:56.148789 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.148786 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nm4v\" (UniqueName: \"kubernetes.io/projected/af60c524-888d-4f3c-8ea0-40899b674491-kube-api-access-7nm4v\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:56.799350 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.799318 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" Oct 30 15:55:56.799776 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.799329 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp" event={"ID":"af60c524-888d-4f3c-8ea0-40899b674491","Type":"ContainerDied","Data":"7e267df9aed59f2b5ec3512780981d2b919e57e9606096bcfb873079212bd563"} Oct 30 15:55:56.799776 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.799441 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e267df9aed59f2b5ec3512780981d2b919e57e9606096bcfb873079212bd563" Oct 30 15:55:56.801202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.801168 4401 generic.go:356] "Generic (PLEG): container finished" podID="9c1b189e-222c-42ee-827c-55653cc72209" containerID="54401bea0a911e95f8683b661f740bb74cd4371eb30738197b51be72ad628246" exitCode=0 Oct 30 15:55:56.801340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:56.801239 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" event={"ID":"9c1b189e-222c-42ee-827c-55653cc72209","Type":"ContainerDied","Data":"54401bea0a911e95f8683b661f740bb74cd4371eb30738197b51be72ad628246"} Oct 30 15:55:57.992161 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:57.992133 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:58.080425 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.080334 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1b189e-222c-42ee-827c-55653cc72209-kube-api-access-fwbkm\" (UniqueName: \"kubernetes.io/projected/9c1b189e-222c-42ee-827c-55653cc72209-kube-api-access-fwbkm\") pod \"9c1b189e-222c-42ee-827c-55653cc72209\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " Oct 30 15:55:58.080600 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.080474 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-xks-ca\") pod \"9c1b189e-222c-42ee-827c-55653cc72209\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " Oct 30 15:55:58.080666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.080642 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-openshift-ca\") pod \"9c1b189e-222c-42ee-827c-55653cc72209\" (UID: \"9c1b189e-222c-42ee-827c-55653cc72209\") " Oct 30 15:55:58.081051 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.081018 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-openshift-ca" pod "9c1b189e-222c-42ee-827c-55653cc72209" (UID: "9c1b189e-222c-42ee-827c-55653cc72209"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:55:58.081355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.081333 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:58.082704 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.082673 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-xks-ca" pod "9c1b189e-222c-42ee-827c-55653cc72209" (UID: "9c1b189e-222c-42ee-827c-55653cc72209"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:55:58.082704 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.082684 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1b189e-222c-42ee-827c-55653cc72209-kube-api-access-fwbkm" pod "9c1b189e-222c-42ee-827c-55653cc72209" (UID: "9c1b189e-222c-42ee-827c-55653cc72209"). InnerVolumeSpecName "kube-api-access-fwbkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:55:58.183288 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.183249 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbkm\" (UniqueName: \"kubernetes.io/projected/9c1b189e-222c-42ee-827c-55653cc72209-kube-api-access-fwbkm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:58.183288 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.183285 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c1b189e-222c-42ee-827c-55653cc72209-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:55:58.820998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.820967 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" Oct 30 15:55:58.820998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.820991 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b" event={"ID":"9c1b189e-222c-42ee-827c-55653cc72209","Type":"ContainerDied","Data":"d341525409de6f62219d1bac90205e6a91eb40e4fd34e45b2accfbe6419444d4"} Oct 30 15:55:58.821236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:55:58.821044 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d341525409de6f62219d1bac90205e6a91eb40e4fd34e45b2accfbe6419444d4" Oct 30 15:56:01.014983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:01.014932 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:01.121032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:01.120968 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:01.163274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:01.163228 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:01.516858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:01.500366 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:56:01.516858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:01.504273 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:01.887913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:01.837041 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:02.065851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.064931 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:02.104854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.082598 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:02.160936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.160263 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:56:02.231870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.223911 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:56:02.363872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.362843 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:02.521912 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:56:02.517781 4401 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.091s" Oct 30 15:56:02.696871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.696197 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:56:02.696871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.696252 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:56:02.696871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.696286 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:56:02.697192 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.600679 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:56:02.713868 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.604292 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:56:02.735531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.735496 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:56:02.735839 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:02.735800 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:56:03.612853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:03.603500 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:03.667890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:03.619314 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:56:03.667890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:03.622190 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:56:03.667890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:03.622982 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:03.667890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:03.633703 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:56:03.701711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:03.699069 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:56:03.804866 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:03.735914 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:04.062505 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.062377 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:56:04.143694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.143646 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:56:04.145811 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.145786 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:56:04.149941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.149914 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:56:04.558572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.554855 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:04.573889 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.573563 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:56:04.583841 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.583025 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:56:04.589918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.584615 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:56:04.589918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.584670 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:56:04.598886 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.594811 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:56:04.977274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.977234 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl"] Oct 30 15:56:04.978228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.978206 4401 cpu_manager.go:414] "containerMap: removing container" podUID="af60c524-888d-4f3c-8ea0-40899b674491" containerName="revoke-keys" Oct 30 15:56:04.978325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.978230 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af60c524-888d-4f3c-8ea0-40899b674491" containerName="revoke-keys" Oct 30 15:56:04.978325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.978264 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9c1b189e-222c-42ee-827c-55653cc72209" containerName="revoke-keys" Oct 30 15:56:04.978325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.978272 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1b189e-222c-42ee-827c-55653cc72209" containerName="revoke-keys" Oct 30 15:56:04.979057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.978507 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="af60c524-888d-4f3c-8ea0-40899b674491" containerName="revoke-keys" Oct 30 15:56:04.979057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.978528 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1b189e-222c-42ee-827c-55653cc72209" containerName="revoke-keys" Oct 30 15:56:04.982922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.982895 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:04.999564 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.999531 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:56:04.999804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.999780 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:56:04.999897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.999731 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:56:04.999897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:04.999666 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:56:05.001970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.001943 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl"] Oct 30 15:56:05.090071 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.090030 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxzv\" (UniqueName: \"kubernetes.io/projected/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-kube-api-access-kqxzv\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.090319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.090296 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.090399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.090360 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.191931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.191889 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.192134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.191960 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.192134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.192089 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxzv\" (UniqueName: \"kubernetes.io/projected/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-kube-api-access-kqxzv\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.193020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.192991 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.194799 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.194772 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.217442 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.217411 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxzv\" (UniqueName: \"kubernetes.io/projected/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-kube-api-access-kqxzv\") pod \"maas-api-revoke-sub-159a0286-nncgl\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.303438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.303346 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:05.671370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:05.671337 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl"] Oct 30 15:56:05.671594 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:05.671570 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3913ff2e_83fd_4a5f_ada0_81d2eb01efaf.slice/crio-b87e5aa08ee105e1176914cb7606672bb89b78f4e8993e936f63fc1d156162ad WatchSource:0}: Error finding container b87e5aa08ee105e1176914cb7606672bb89b78f4e8993e936f63fc1d156162ad: Status 404 returned error can't find the container with id b87e5aa08ee105e1176914cb7606672bb89b78f4e8993e936f63fc1d156162ad Oct 30 15:56:06.070845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:06.070791 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" event={"ID":"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf","Type":"ContainerStarted","Data":"f5f894d0af9bf3f121043180408e3cc0e5764084e0af7930694ff105e3dfb736"} Oct 30 15:56:06.071276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:06.070897 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" event={"ID":"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf","Type":"ContainerStarted","Data":"b87e5aa08ee105e1176914cb7606672bb89b78f4e8993e936f63fc1d156162ad"} Oct 30 15:56:06.117121 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:06.117063 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" podStartSLOduration=2.117047488 podStartE2EDuration="2.117047488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:06.114148116 +0000 UTC m=+1074.813421274" watchObservedRunningTime="2026-10-30 15:56:06.117047488 +0000 UTC m=+1074.816320645" Oct 30 15:56:07.080625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:07.080589 4401 generic.go:356] "Generic (PLEG): container finished" podID="3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" containerID="f5f894d0af9bf3f121043180408e3cc0e5764084e0af7930694ff105e3dfb736" exitCode=0 Oct 30 15:56:07.081085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:07.080684 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" event={"ID":"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf","Type":"ContainerDied","Data":"f5f894d0af9bf3f121043180408e3cc0e5764084e0af7930694ff105e3dfb736"} Oct 30 15:56:08.265084 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.265054 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:08.347169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.347056 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-xks-ca\") pod \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " Oct 30 15:56:08.347169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.347120 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-openshift-ca\") pod \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " Oct 30 15:56:08.347438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.347245 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-kube-api-access-kqxzv\" (UniqueName: \"kubernetes.io/projected/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-kube-api-access-kqxzv\") pod \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\" (UID: \"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf\") " Oct 30 15:56:08.347554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.347522 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-openshift-ca" pod "3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" (UID: "3913ff2e-83fd-4a5f-ada0-81d2eb01efaf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:08.347808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.347783 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:08.349426 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.349396 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-xks-ca" pod "3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" (UID: "3913ff2e-83fd-4a5f-ada0-81d2eb01efaf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:08.349608 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.349579 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-kube-api-access-kqxzv" pod "3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" (UID: "3913ff2e-83fd-4a5f-ada0-81d2eb01efaf"). InnerVolumeSpecName "kube-api-access-kqxzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:08.449520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.449471 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqxzv\" (UniqueName: \"kubernetes.io/projected/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-kube-api-access-kqxzv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:08.449520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:08.449509 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:09.102031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:09.101998 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" event={"ID":"3913ff2e-83fd-4a5f-ada0-81d2eb01efaf","Type":"ContainerDied","Data":"b87e5aa08ee105e1176914cb7606672bb89b78f4e8993e936f63fc1d156162ad"} Oct 30 15:56:09.102031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:09.102031 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl" Oct 30 15:56:09.102276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:09.102048 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b87e5aa08ee105e1176914cb7606672bb89b78f4e8993e936f63fc1d156162ad" Oct 30 15:56:10.103625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.103589 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm"] Oct 30 15:56:10.104577 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.104551 4401 cpu_manager.go:414] "containerMap: removing container" podUID="3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" containerName="revoke-keys" Oct 30 15:56:10.104684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.104578 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" containerName="revoke-keys" Oct 30 15:56:10.104774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.104759 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" containerName="revoke-keys" Oct 30 15:56:10.109056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.109032 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.130969 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.130928 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm"] Oct 30 15:56:10.137877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.137808 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:56:10.137877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.137808 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:56:10.138096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.137859 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:56:10.140547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.140524 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:56:10.285000 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.284962 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.285183 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.285148 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.285337 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.285315 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hwt\" (UniqueName: \"kubernetes.io/projected/7772b8a2-b1a1-484e-ba72-7d5ec5662def-kube-api-access-t4hwt\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.387256 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.387164 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.387413 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.387285 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hwt\" (UniqueName: \"kubernetes.io/projected/7772b8a2-b1a1-484e-ba72-7d5ec5662def-kube-api-access-t4hwt\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.387413 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.387357 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.388081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.388041 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.389593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.389566 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.412761 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.412732 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hwt\" (UniqueName: \"kubernetes.io/projected/7772b8a2-b1a1-484e-ba72-7d5ec5662def-kube-api-access-t4hwt\") pod \"maas-api-revoke-sub-34844f41-zl4vm\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.424382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.424352 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:10.794995 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:10.794298 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm"] Oct 30 15:56:11.120462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:11.120420 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" event={"ID":"7772b8a2-b1a1-484e-ba72-7d5ec5662def","Type":"ContainerStarted","Data":"65340d35734551a9862b2bd63e857277e2bb9ca73a7797c4d3a72f09df2a8e05"} Oct 30 15:56:12.131891 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:12.131817 4401 generic.go:356] "Generic (PLEG): container finished" podID="7772b8a2-b1a1-484e-ba72-7d5ec5662def" containerID="ced473035fb61d2e525e1a4e5aad2db37de180d2b8362f3ca32fe778977bcd8e" exitCode=0 Oct 30 15:56:12.132330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:12.131913 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" event={"ID":"7772b8a2-b1a1-484e-ba72-7d5ec5662def","Type":"ContainerDied","Data":"ced473035fb61d2e525e1a4e5aad2db37de180d2b8362f3ca32fe778977bcd8e"} Oct 30 15:56:13.214891 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.214854 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:56:13.316745 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.316715 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:13.357530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.357492 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-openshift-ca\") pod \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " Oct 30 15:56:13.357755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.357610 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-xks-ca\") pod \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " Oct 30 15:56:13.357755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.357673 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7772b8a2-b1a1-484e-ba72-7d5ec5662def-kube-api-access-t4hwt\" (UniqueName: \"kubernetes.io/projected/7772b8a2-b1a1-484e-ba72-7d5ec5662def-kube-api-access-t4hwt\") pod \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\" (UID: \"7772b8a2-b1a1-484e-ba72-7d5ec5662def\") " Oct 30 15:56:13.358020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.357996 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-openshift-ca" pod "7772b8a2-b1a1-484e-ba72-7d5ec5662def" (UID: "7772b8a2-b1a1-484e-ba72-7d5ec5662def"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:13.358318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.358299 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:13.359918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.359886 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-xks-ca" pod "7772b8a2-b1a1-484e-ba72-7d5ec5662def" (UID: "7772b8a2-b1a1-484e-ba72-7d5ec5662def"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:13.360165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.360141 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7772b8a2-b1a1-484e-ba72-7d5ec5662def-kube-api-access-t4hwt" pod "7772b8a2-b1a1-484e-ba72-7d5ec5662def" (UID: "7772b8a2-b1a1-484e-ba72-7d5ec5662def"). InnerVolumeSpecName "kube-api-access-t4hwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:13.459876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.459817 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7772b8a2-b1a1-484e-ba72-7d5ec5662def-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:13.459876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:13.459877 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hwt\" (UniqueName: \"kubernetes.io/projected/7772b8a2-b1a1-484e-ba72-7d5ec5662def-kube-api-access-t4hwt\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:14.252489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:14.252444 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" event={"ID":"7772b8a2-b1a1-484e-ba72-7d5ec5662def","Type":"ContainerDied","Data":"65340d35734551a9862b2bd63e857277e2bb9ca73a7797c4d3a72f09df2a8e05"} Oct 30 15:56:14.254156 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:14.254116 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65340d35734551a9862b2bd63e857277e2bb9ca73a7797c4d3a72f09df2a8e05" Oct 30 15:56:14.254374 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:14.253998 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm" Oct 30 15:56:16.589852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:16.588926 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:16.595986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:16.595930 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:16.606998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:16.606948 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:16.607653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:16.607620 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:56:16.634043 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:16.633981 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:17.022996 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.022941 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:17.023354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.023323 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:56:17.327911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.327365 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:17.622059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.622013 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:17.631504 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.631457 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:56:17.691699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.636972 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:56:17.691699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.637019 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:56:17.691699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.637042 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:56:17.705882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.704878 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:56:17.705882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.704946 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:56:17.705882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.704980 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:56:17.705882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.705012 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:56:17.705882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.705046 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:56:17.917148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:17.916816 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:18.671852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.659938 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:56:18.682317 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.682270 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:18.733846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.683913 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:56:18.733846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.685578 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:18.810842 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.806628 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:18.838850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.837325 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:56:18.869848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.865339 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:56:18.908986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.908950 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:56:18.944936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.933573 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:56:18.944936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.934520 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:56:18.959069 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:18.959023 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:19.145856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:19.108538 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:19.197909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:19.187353 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 15:56:19.336313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:19.331534 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:56:19.336313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:19.332400 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:56:19.338530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:19.338498 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:56:22.025042 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:56:22.024991 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:56:22.977007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:22.976970 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7"] Oct 30 15:56:22.977678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:22.977665 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7772b8a2-b1a1-484e-ba72-7d5ec5662def" containerName="revoke-keys" Oct 30 15:56:22.977725 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:22.977680 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7772b8a2-b1a1-484e-ba72-7d5ec5662def" containerName="revoke-keys" Oct 30 15:56:22.977801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:22.977792 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7772b8a2-b1a1-484e-ba72-7d5ec5662def" containerName="revoke-keys" Oct 30 15:56:22.981652 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:22.981630 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.002933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.002891 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7"] Oct 30 15:56:23.006457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.006426 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:56:23.006684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.006518 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:56:23.006684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.006551 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:56:23.012257 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.012229 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:56:23.031301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.031229 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.031875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.031339 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.032092 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.032063 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplnm\" (UniqueName: \"kubernetes.io/projected/b6ed217d-339e-45bb-ac01-3db35e7b8b90-kube-api-access-dplnm\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.133589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.133546 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.133768 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.133602 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.133768 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.133740 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplnm\" (UniqueName: \"kubernetes.io/projected/b6ed217d-339e-45bb-ac01-3db35e7b8b90-kube-api-access-dplnm\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.134358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.134337 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.136152 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.136127 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.159335 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.159296 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplnm\" (UniqueName: \"kubernetes.io/projected/b6ed217d-339e-45bb-ac01-3db35e7b8b90-kube-api-access-dplnm\") pod \"maas-api-revoke-sub-07a0c838-lhsw7\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.301452 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.301354 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:23.615656 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:23.615623 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ed217d_339e_45bb_ac01_3db35e7b8b90.slice/crio-9c1c70465671eff58bad6ea00c33c97047adeed24ed14155c6fb10ac752edb2c WatchSource:0}: Error finding container 9c1c70465671eff58bad6ea00c33c97047adeed24ed14155c6fb10ac752edb2c: Status 404 returned error can't find the container with id 9c1c70465671eff58bad6ea00c33c97047adeed24ed14155c6fb10ac752edb2c Oct 30 15:56:23.616879 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.616846 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7"] Oct 30 15:56:23.951936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.951906 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn"] Oct 30 15:56:23.957739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.957715 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:23.977598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:23.977555 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn"] Oct 30 15:56:24.102423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.102367 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.102918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.102442 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.102918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.102664 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbvn\" (UniqueName: \"kubernetes.io/projected/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-kube-api-access-npbvn\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.204387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.204291 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npbvn\" (UniqueName: \"kubernetes.io/projected/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-kube-api-access-npbvn\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.204551 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.204472 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.204551 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.204518 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.205277 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.205255 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.207000 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.206975 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.229944 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.229915 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbvn\" (UniqueName: \"kubernetes.io/projected/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-kube-api-access-npbvn\") pod \"maas-api-revoke-sub-cff3d958-ppskn\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.275872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.275807 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:24.424524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.424481 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" event={"ID":"b6ed217d-339e-45bb-ac01-3db35e7b8b90","Type":"ContainerStarted","Data":"5b7b8f88365d207ff3e3a9a7eb8373972ae26a4d4164bb5849a6a6bbde2df7d2"} Oct 30 15:56:24.424747 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.424548 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" event={"ID":"b6ed217d-339e-45bb-ac01-3db35e7b8b90","Type":"ContainerStarted","Data":"9c1c70465671eff58bad6ea00c33c97047adeed24ed14155c6fb10ac752edb2c"} Oct 30 15:56:24.469226 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.469122 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" podStartSLOduration=2.469107874 podStartE2EDuration="2.469107874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:24.468624439 +0000 UTC m=+1093.167897599" watchObservedRunningTime="2026-10-30 15:56:24.469107874 +0000 UTC m=+1093.168381064" Oct 30 15:56:24.593084 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:24.593050 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ace5cb_3ee4_4368_933f_3cd5ea1f92a0.slice/crio-34fca40d9d5743b24b820142b4b327c089e807b482ffa66435503fa7232e4308 WatchSource:0}: Error finding container 34fca40d9d5743b24b820142b4b327c089e807b482ffa66435503fa7232e4308: Status 404 returned error can't find the container with id 34fca40d9d5743b24b820142b4b327c089e807b482ffa66435503fa7232e4308 Oct 30 15:56:24.596595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:24.596564 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn"] Oct 30 15:56:25.256549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.256516 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658"] Oct 30 15:56:25.261396 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.261370 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.279462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.279426 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658"] Oct 30 15:56:25.421377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.421276 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.421377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.421318 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjjt\" (UniqueName: \"kubernetes.io/projected/8320b0ca-66cc-45db-b79c-2e18812e474c-kube-api-access-tjjjt\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.421615 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.421596 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.435682 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.435648 4401 generic.go:356] "Generic (PLEG): container finished" podID="b6ed217d-339e-45bb-ac01-3db35e7b8b90" containerID="5b7b8f88365d207ff3e3a9a7eb8373972ae26a4d4164bb5849a6a6bbde2df7d2" exitCode=0 Oct 30 15:56:25.435890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.435722 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" event={"ID":"b6ed217d-339e-45bb-ac01-3db35e7b8b90","Type":"ContainerDied","Data":"5b7b8f88365d207ff3e3a9a7eb8373972ae26a4d4164bb5849a6a6bbde2df7d2"} Oct 30 15:56:25.438044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.438003 4401 generic.go:356] "Generic (PLEG): container finished" podID="c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" containerID="132b037633f52aba163bef69b1e2be254b6d6734950bbd2e5df48d9c6256ee43" exitCode=0 Oct 30 15:56:25.438180 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.438143 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" event={"ID":"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0","Type":"ContainerDied","Data":"132b037633f52aba163bef69b1e2be254b6d6734950bbd2e5df48d9c6256ee43"} Oct 30 15:56:25.438239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.438220 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" event={"ID":"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0","Type":"ContainerStarted","Data":"34fca40d9d5743b24b820142b4b327c089e807b482ffa66435503fa7232e4308"} Oct 30 15:56:25.536585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.536546 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.548451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.548408 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjjt\" (UniqueName: \"kubernetes.io/projected/8320b0ca-66cc-45db-b79c-2e18812e474c-kube-api-access-tjjjt\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.551566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.551521 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.552445 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.552417 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.555762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.554301 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.577993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.577957 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjjt\" (UniqueName: \"kubernetes.io/projected/8320b0ca-66cc-45db-b79c-2e18812e474c-kube-api-access-tjjjt\") pod \"maas-api-revoke-sub-1cf22e7a-pm658\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.585299 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.585266 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:25.923116 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:25.923046 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8320b0ca_66cc_45db_b79c_2e18812e474c.slice/crio-30705902834c9178d29f72a8ad93b4ae820d07269212dc38f80c8a5ee4f804d7 WatchSource:0}: Error finding container 30705902834c9178d29f72a8ad93b4ae820d07269212dc38f80c8a5ee4f804d7: Status 404 returned error can't find the container with id 30705902834c9178d29f72a8ad93b4ae820d07269212dc38f80c8a5ee4f804d7 Oct 30 15:56:25.924874 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:25.924773 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658"] Oct 30 15:56:26.448582 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.448544 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" event={"ID":"8320b0ca-66cc-45db-b79c-2e18812e474c","Type":"ContainerStarted","Data":"18e0cab785f8096477fcd2444fe58fb93c41160d7b3d4b7f0324d9ddae1eb064"} Oct 30 15:56:26.449107 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.448601 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" event={"ID":"8320b0ca-66cc-45db-b79c-2e18812e474c","Type":"ContainerStarted","Data":"30705902834c9178d29f72a8ad93b4ae820d07269212dc38f80c8a5ee4f804d7"} Oct 30 15:56:26.494087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.494029 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" podStartSLOduration=1.494013024 podStartE2EDuration="1.494013024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:26.492866039 +0000 UTC m=+1095.192139192" watchObservedRunningTime="2026-10-30 15:56:26.494013024 +0000 UTC m=+1095.193286180" Oct 30 15:56:26.626935 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.626851 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:26.667556 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.667523 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ed217d-339e-45bb-ac01-3db35e7b8b90-kube-api-access-dplnm\" (UniqueName: \"kubernetes.io/projected/b6ed217d-339e-45bb-ac01-3db35e7b8b90-kube-api-access-dplnm\") pod \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " Oct 30 15:56:26.667753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.667622 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-xks-ca\") pod \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " Oct 30 15:56:26.667753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.667665 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-openshift-ca\") pod \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\" (UID: \"b6ed217d-339e-45bb-ac01-3db35e7b8b90\") " Oct 30 15:56:26.668399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.668363 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-openshift-ca" pod "b6ed217d-339e-45bb-ac01-3db35e7b8b90" (UID: "b6ed217d-339e-45bb-ac01-3db35e7b8b90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:26.681026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.672046 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:26.681026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.674293 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-xks-ca" pod "b6ed217d-339e-45bb-ac01-3db35e7b8b90" (UID: "b6ed217d-339e-45bb-ac01-3db35e7b8b90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:26.681026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.674685 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ed217d-339e-45bb-ac01-3db35e7b8b90-kube-api-access-dplnm" pod "b6ed217d-339e-45bb-ac01-3db35e7b8b90" (UID: "b6ed217d-339e-45bb-ac01-3db35e7b8b90"). InnerVolumeSpecName "kube-api-access-dplnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:26.700428 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.700395 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:26.772604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.772565 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-openshift-ca\") pod \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " Oct 30 15:56:26.772813 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.772686 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-xks-ca\") pod \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " Oct 30 15:56:26.772813 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.772721 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-kube-api-access-npbvn\" (UniqueName: \"kubernetes.io/projected/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-kube-api-access-npbvn\") pod \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\" (UID: \"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0\") " Oct 30 15:56:26.773092 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.773060 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-openshift-ca" pod "c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" (UID: "c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:26.773630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.773601 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6ed217d-339e-45bb-ac01-3db35e7b8b90-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:26.773737 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.773633 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:26.773737 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.773652 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dplnm\" (UniqueName: \"kubernetes.io/projected/b6ed217d-339e-45bb-ac01-3db35e7b8b90-kube-api-access-dplnm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:26.775135 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.775098 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-kube-api-access-npbvn" pod "c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" (UID: "c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0"). InnerVolumeSpecName "kube-api-access-npbvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:26.775389 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.775363 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-xks-ca" pod "c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" (UID: "c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:26.875104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.875077 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:26.875104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:26.875107 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npbvn\" (UniqueName: \"kubernetes.io/projected/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0-kube-api-access-npbvn\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:27.459646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.459603 4401 generic.go:356] "Generic (PLEG): container finished" podID="8320b0ca-66cc-45db-b79c-2e18812e474c" containerID="18e0cab785f8096477fcd2444fe58fb93c41160d7b3d4b7f0324d9ddae1eb064" exitCode=0 Oct 30 15:56:27.460078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.459668 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" event={"ID":"8320b0ca-66cc-45db-b79c-2e18812e474c","Type":"ContainerDied","Data":"18e0cab785f8096477fcd2444fe58fb93c41160d7b3d4b7f0324d9ddae1eb064"} Oct 30 15:56:27.461602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.461580 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" Oct 30 15:56:27.461759 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.461603 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7" event={"ID":"b6ed217d-339e-45bb-ac01-3db35e7b8b90","Type":"ContainerDied","Data":"9c1c70465671eff58bad6ea00c33c97047adeed24ed14155c6fb10ac752edb2c"} Oct 30 15:56:27.461759 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.461652 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c1c70465671eff58bad6ea00c33c97047adeed24ed14155c6fb10ac752edb2c" Oct 30 15:56:27.463333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.463308 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" Oct 30 15:56:27.463489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.463335 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn" event={"ID":"c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0","Type":"ContainerDied","Data":"34fca40d9d5743b24b820142b4b327c089e807b482ffa66435503fa7232e4308"} Oct 30 15:56:27.463489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:27.463393 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34fca40d9d5743b24b820142b4b327c089e807b482ffa66435503fa7232e4308" Oct 30 15:56:28.654939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.654910 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:28.700311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.700269 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-openshift-ca\") pod \"8320b0ca-66cc-45db-b79c-2e18812e474c\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " Oct 30 15:56:28.700783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.700648 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-openshift-ca" pod "8320b0ca-66cc-45db-b79c-2e18812e474c" (UID: "8320b0ca-66cc-45db-b79c-2e18812e474c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:28.701037 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.701013 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-xks-ca\") pod \"8320b0ca-66cc-45db-b79c-2e18812e474c\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " Oct 30 15:56:28.701164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.701067 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8320b0ca-66cc-45db-b79c-2e18812e474c-kube-api-access-tjjjt\" (UniqueName: \"kubernetes.io/projected/8320b0ca-66cc-45db-b79c-2e18812e474c-kube-api-access-tjjjt\") pod \"8320b0ca-66cc-45db-b79c-2e18812e474c\" (UID: \"8320b0ca-66cc-45db-b79c-2e18812e474c\") " Oct 30 15:56:28.701689 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.701621 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:28.704590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.704539 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8320b0ca-66cc-45db-b79c-2e18812e474c-kube-api-access-tjjjt" pod "8320b0ca-66cc-45db-b79c-2e18812e474c" (UID: "8320b0ca-66cc-45db-b79c-2e18812e474c"). InnerVolumeSpecName "kube-api-access-tjjjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:28.704714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.704631 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-xks-ca" pod "8320b0ca-66cc-45db-b79c-2e18812e474c" (UID: "8320b0ca-66cc-45db-b79c-2e18812e474c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:28.803638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.803540 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8320b0ca-66cc-45db-b79c-2e18812e474c-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:28.803638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:28.803605 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjjjt\" (UniqueName: \"kubernetes.io/projected/8320b0ca-66cc-45db-b79c-2e18812e474c-kube-api-access-tjjjt\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:29.484353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:29.484319 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" Oct 30 15:56:29.484531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:29.484348 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658" event={"ID":"8320b0ca-66cc-45db-b79c-2e18812e474c","Type":"ContainerDied","Data":"30705902834c9178d29f72a8ad93b4ae820d07269212dc38f80c8a5ee4f804d7"} Oct 30 15:56:29.484531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:29.484400 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30705902834c9178d29f72a8ad93b4ae820d07269212dc38f80c8a5ee4f804d7" Oct 30 15:56:30.797656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.797616 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr"] Oct 30 15:56:30.798477 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798453 4401 cpu_manager.go:414] "containerMap: removing container" podUID="c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" containerName="revoke-keys" Oct 30 15:56:30.798477 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798475 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" containerName="revoke-keys" Oct 30 15:56:30.798638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798494 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b6ed217d-339e-45bb-ac01-3db35e7b8b90" containerName="revoke-keys" Oct 30 15:56:30.798638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798500 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ed217d-339e-45bb-ac01-3db35e7b8b90" containerName="revoke-keys" Oct 30 15:56:30.798638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798509 4401 cpu_manager.go:414] "containerMap: removing container" podUID="8320b0ca-66cc-45db-b79c-2e18812e474c" containerName="revoke-keys" Oct 30 15:56:30.798638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798514 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8320b0ca-66cc-45db-b79c-2e18812e474c" containerName="revoke-keys" Oct 30 15:56:30.798638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798629 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ed217d-339e-45bb-ac01-3db35e7b8b90" containerName="revoke-keys" Oct 30 15:56:30.798638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798639 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="8320b0ca-66cc-45db-b79c-2e18812e474c" containerName="revoke-keys" Oct 30 15:56:30.798818 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.798654 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" containerName="revoke-keys" Oct 30 15:56:30.802662 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.802622 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.822801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.822762 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr"] Oct 30 15:56:30.827531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.827500 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:56:30.827531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.827524 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:56:30.827749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.827550 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:56:30.827749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.827559 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:56:30.838058 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.838020 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprck\" (UniqueName: \"kubernetes.io/projected/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-kube-api-access-vprck\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.838255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.838224 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.838362 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.838341 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.939718 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.939677 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.940184 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.940154 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprck\" (UniqueName: \"kubernetes.io/projected/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-kube-api-access-vprck\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.940311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.940284 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.941015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.940992 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.942973 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.942948 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:30.965928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:30.965890 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprck\" (UniqueName: \"kubernetes.io/projected/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-kube-api-access-vprck\") pod \"maas-api-revoke-sub-43d43400-9scfr\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:31.126341 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:31.126310 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:31.440440 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:31.440406 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfb530f_dfd4_45ab_b7cd_03ea3459c3c2.slice/crio-8c79251ee5d6c89f531d110d2c9cefa0083512359826ef958c70a732d16b1796 WatchSource:0}: Error finding container 8c79251ee5d6c89f531d110d2c9cefa0083512359826ef958c70a732d16b1796: Status 404 returned error can't find the container with id 8c79251ee5d6c89f531d110d2c9cefa0083512359826ef958c70a732d16b1796 Oct 30 15:56:31.442048 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:31.442016 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr"] Oct 30 15:56:31.504282 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:31.504237 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" event={"ID":"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2","Type":"ContainerStarted","Data":"8c79251ee5d6c89f531d110d2c9cefa0083512359826ef958c70a732d16b1796"} Oct 30 15:56:32.514932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:32.514891 4401 generic.go:356] "Generic (PLEG): container finished" podID="3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" containerID="981c103ae38a263f3b2629f99407843057f62727a0256f2c0898f6b267045f7f" exitCode=0 Oct 30 15:56:32.515336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:32.514922 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" event={"ID":"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2","Type":"ContainerDied","Data":"981c103ae38a263f3b2629f99407843057f62727a0256f2c0898f6b267045f7f"} Oct 30 15:56:34.522914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.521198 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:34.682904 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.678761 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-openshift-ca\") pod \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " Oct 30 15:56:34.705870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.688315 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-openshift-ca" pod "3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" (UID: "3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:34.705870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.678818 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-xks-ca\") pod \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " Oct 30 15:56:34.705870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.690247 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-kube-api-access-vprck\" (UniqueName: \"kubernetes.io/projected/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-kube-api-access-vprck\") pod \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\" (UID: \"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2\") " Oct 30 15:56:34.705870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.691053 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:34.715916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.714098 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-kube-api-access-vprck" pod "3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" (UID: "3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2"). InnerVolumeSpecName "kube-api-access-vprck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:34.748852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.728383 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-xks-ca" pod "3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" (UID: "3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:34.901854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.898783 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vprck\" (UniqueName: \"kubernetes.io/projected/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-kube-api-access-vprck\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:34.901854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.898843 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:34.911868 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.909244 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb"] Oct 30 15:56:34.911868 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.910326 4401 cpu_manager.go:414] "containerMap: removing container" podUID="3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" containerName="revoke-keys" Oct 30 15:56:34.911868 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.910352 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" containerName="revoke-keys" Oct 30 15:56:34.911868 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.910617 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" containerName="revoke-keys" Oct 30 15:56:34.985890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:34.973099 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.187850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.184691 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" event={"ID":"3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2","Type":"ContainerDied","Data":"8c79251ee5d6c89f531d110d2c9cefa0083512359826ef958c70a732d16b1796"} Oct 30 15:56:35.188152 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.188119 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c79251ee5d6c89f531d110d2c9cefa0083512359826ef958c70a732d16b1796" Oct 30 15:56:35.188393 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.188372 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr" Oct 30 15:56:35.220929 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.220881 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.221217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.221193 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.221489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.221469 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pgwg\" (UniqueName: \"kubernetes.io/projected/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-kube-api-access-7pgwg\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.320880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.319501 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb"] Oct 30 15:56:35.386946 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.355447 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:35.388860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.388797 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.389175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.389149 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.389553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.389517 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pgwg\" (UniqueName: \"kubernetes.io/projected/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-kube-api-access-7pgwg\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.393867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.391424 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.609022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.582435 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pgwg\" (UniqueName: \"kubernetes.io/projected/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-kube-api-access-7pgwg\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.609022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.606732 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-448e74a3-q8lnb\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.661679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.659299 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:35.835102 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.806012 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:35.835102 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.807256 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:56:35.887988 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.887928 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:56:35.913870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.900500 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k"] Oct 30 15:56:35.946861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:35.945961 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:56:36.020143 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.020107 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k"] Oct 30 15:56:36.045990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.035087 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.107850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.104127 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:56:36.119436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.110346 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.120046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.120007 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.120352 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.120331 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2fb\" (UniqueName: \"kubernetes.io/projected/a92f9022-bc45-4137-80ee-b0f1b36abae7-kube-api-access-lq2fb\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.235189 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.235150 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2fb\" (UniqueName: \"kubernetes.io/projected/a92f9022-bc45-4137-80ee-b0f1b36abae7-kube-api-access-lq2fb\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.235523 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.235498 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.235920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.235893 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.237328 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.237299 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.277853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.274653 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2fb\" (UniqueName: \"kubernetes.io/projected/a92f9022-bc45-4137-80ee-b0f1b36abae7-kube-api-access-lq2fb\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.285483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.285444 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c40e426b-ftz4k\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.368911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.366211 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:56:36.368911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.366282 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:36.368911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.366612 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:56:36.368911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.367021 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:56:36.374197 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.373812 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:56:36.562556 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.562513 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:36.763414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.752938 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:56:36.763414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.753007 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:36.763414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.753424 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:56:36.763414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.753480 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:36.853305 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.834753 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:36.853305 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.835153 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:56:36.853305 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.835469 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:56:36.853305 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.835529 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:56:36.976915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.929918 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:56:36.976915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.929985 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:36.976915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.931948 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:36.976915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.932364 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:56:36.976915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.968976 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:36.976915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:36.969491 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:56:37.045382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.044911 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 15:56:37.213006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.211900 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:37.213006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.212177 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:56:37.375047 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.374974 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:37.402021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.399271 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:37.856864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.836131 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:37.856864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.837659 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:37.865803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.863005 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:56:37.865803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.863176 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:37.940533 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.940478 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:37.976991 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:37.976940 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:38.117302 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:38.117183 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26631d06_17b9_4ce8_80d5_ab4dcc21cf19.slice/crio-26f4a81cd9d615686bd86cfa1b77111d5273e6c787419c725d88efb161ee8544 WatchSource:0}: Error finding container 26f4a81cd9d615686bd86cfa1b77111d5273e6c787419c725d88efb161ee8544: Status 404 returned error can't find the container with id 26f4a81cd9d615686bd86cfa1b77111d5273e6c787419c725d88efb161ee8544 Oct 30 15:56:38.212413 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.212361 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:38.394853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.373952 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:56:38.408863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.408528 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:56:38.433318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.433284 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:56:38.854859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.836394 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:56:38.859494 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.857801 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:56:38.871898 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.869474 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:56:38.874425 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.873404 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:38.928044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.927921 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb"] Oct 30 15:56:38.952136 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.949937 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:56:38.959567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.958867 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" event={"ID":"26631d06-17b9-4ce8-80d5-ab4dcc21cf19","Type":"ContainerStarted","Data":"26f4a81cd9d615686bd86cfa1b77111d5273e6c787419c725d88efb161ee8544"} Oct 30 15:56:38.996996 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:38.996931 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:39.272929 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.259210 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:39.387106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.387070 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:56:39.388260 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.388220 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 15:56:39.390716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.388437 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:39.391853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.391811 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:56:39.395162 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.395134 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:56:39.436665 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.436579 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:56:39.451161 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.450719 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:56:39.482502 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.482474 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:56:39.482654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.482512 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k"] Oct 30 15:56:39.797398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.797361 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg"] Oct 30 15:56:39.804682 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.804649 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.829012 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.828979 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg"] Oct 30 15:56:39.862727 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.862600 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.862727 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.862664 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7gb\" (UniqueName: \"kubernetes.io/projected/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-kube-api-access-5w7gb\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.863259 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.863111 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.964576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.964537 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.964695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.964620 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.964736 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.964691 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7gb\" (UniqueName: \"kubernetes.io/projected/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-kube-api-access-5w7gb\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.965300 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.965273 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.967269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.967244 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:39.971215 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.971188 4401 generic.go:356] "Generic (PLEG): container finished" podID="26631d06-17b9-4ce8-80d5-ab4dcc21cf19" containerID="e17a1df9c08181682feee4e6b37b7714db8775a62c5ce1a2c7a4d0a0749cc582" exitCode=0 Oct 30 15:56:39.971338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.971272 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" event={"ID":"26631d06-17b9-4ce8-80d5-ab4dcc21cf19","Type":"ContainerDied","Data":"e17a1df9c08181682feee4e6b37b7714db8775a62c5ce1a2c7a4d0a0749cc582"} Oct 30 15:56:39.973108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.973081 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" event={"ID":"a92f9022-bc45-4137-80ee-b0f1b36abae7","Type":"ContainerStarted","Data":"e6fae34d6ab616cc2f4cef8afae62cdd33c1b9014bfbd44e282ddce7e657edb1"} Oct 30 15:56:39.973190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.973130 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" event={"ID":"a92f9022-bc45-4137-80ee-b0f1b36abae7","Type":"ContainerStarted","Data":"ea5f0bd2c184fa39fcde3f6e9d8d6197975184a6ce4c7ff72e8289af3c2a2667"} Oct 30 15:56:39.990284 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:39.990253 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7gb\" (UniqueName: \"kubernetes.io/projected/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-kube-api-access-5w7gb\") pod \"maas-api-revoke-sub-0a4324a5-m4mpg\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:40.113601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:40.113526 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" podStartSLOduration=6.113503075 podStartE2EDuration="6.113503075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:40.096868955 +0000 UTC m=+1108.796142113" watchObservedRunningTime="2026-10-30 15:56:40.113503075 +0000 UTC m=+1108.812776235" Oct 30 15:56:40.131781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:40.131736 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:40.489115 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:40.489074 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a04bd0_7ab3_450f_9139_3ab5d8993aa7.slice/crio-01d81fbeb02010b03241ce7cb09fcd5e19c3413d468df4f10b03e12191b6bd82 WatchSource:0}: Error finding container 01d81fbeb02010b03241ce7cb09fcd5e19c3413d468df4f10b03e12191b6bd82: Status 404 returned error can't find the container with id 01d81fbeb02010b03241ce7cb09fcd5e19c3413d468df4f10b03e12191b6bd82 Oct 30 15:56:40.491262 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:40.491238 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg"] Oct 30 15:56:40.984418 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:40.984384 4401 generic.go:356] "Generic (PLEG): container finished" podID="a92f9022-bc45-4137-80ee-b0f1b36abae7" containerID="e6fae34d6ab616cc2f4cef8afae62cdd33c1b9014bfbd44e282ddce7e657edb1" exitCode=0 Oct 30 15:56:40.984949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:40.984476 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" event={"ID":"a92f9022-bc45-4137-80ee-b0f1b36abae7","Type":"ContainerDied","Data":"e6fae34d6ab616cc2f4cef8afae62cdd33c1b9014bfbd44e282ddce7e657edb1"} Oct 30 15:56:40.986423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:40.986394 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" event={"ID":"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7","Type":"ContainerStarted","Data":"da11bd58f4339614243e2959b65e56d23fedb8fea1a86c23437a92596ac6e7ae"} Oct 30 15:56:40.986552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:40.986446 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" event={"ID":"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7","Type":"ContainerStarted","Data":"01d81fbeb02010b03241ce7cb09fcd5e19c3413d468df4f10b03e12191b6bd82"} Oct 30 15:56:41.073470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.073399 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" podStartSLOduration=2.073380743 podStartE2EDuration="2.073380743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:41.071625244 +0000 UTC m=+1109.770898403" watchObservedRunningTime="2026-10-30 15:56:41.073380743 +0000 UTC m=+1109.772653902" Oct 30 15:56:41.142276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.142246 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:41.185013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.184975 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-xks-ca\") pod \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " Oct 30 15:56:41.185214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.185116 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-openshift-ca\") pod \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " Oct 30 15:56:41.185295 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.185235 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-kube-api-access-7pgwg\" (UniqueName: \"kubernetes.io/projected/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-kube-api-access-7pgwg\") pod \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\" (UID: \"26631d06-17b9-4ce8-80d5-ab4dcc21cf19\") " Oct 30 15:56:41.185605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.185573 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-openshift-ca" pod "26631d06-17b9-4ce8-80d5-ab4dcc21cf19" (UID: "26631d06-17b9-4ce8-80d5-ab4dcc21cf19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:41.186780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.186749 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:41.188029 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.187997 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-xks-ca" pod "26631d06-17b9-4ce8-80d5-ab4dcc21cf19" (UID: "26631d06-17b9-4ce8-80d5-ab4dcc21cf19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:41.188483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.188459 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-kube-api-access-7pgwg" pod "26631d06-17b9-4ce8-80d5-ab4dcc21cf19" (UID: "26631d06-17b9-4ce8-80d5-ab4dcc21cf19"). InnerVolumeSpecName "kube-api-access-7pgwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:41.288229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.288185 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:41.288229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.288222 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pgwg\" (UniqueName: \"kubernetes.io/projected/26631d06-17b9-4ce8-80d5-ab4dcc21cf19-kube-api-access-7pgwg\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:41.997801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.997699 4401 generic.go:356] "Generic (PLEG): container finished" podID="f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" containerID="da11bd58f4339614243e2959b65e56d23fedb8fea1a86c23437a92596ac6e7ae" exitCode=0 Oct 30 15:56:41.997801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.997781 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" event={"ID":"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7","Type":"ContainerDied","Data":"da11bd58f4339614243e2959b65e56d23fedb8fea1a86c23437a92596ac6e7ae"} Oct 30 15:56:41.999572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.999552 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" Oct 30 15:56:41.999572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.999554 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb" event={"ID":"26631d06-17b9-4ce8-80d5-ab4dcc21cf19","Type":"ContainerDied","Data":"26f4a81cd9d615686bd86cfa1b77111d5273e6c787419c725d88efb161ee8544"} Oct 30 15:56:41.999720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:41.999605 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f4a81cd9d615686bd86cfa1b77111d5273e6c787419c725d88efb161ee8544" Oct 30 15:56:42.194472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.194439 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:42.303303 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.303207 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92f9022-bc45-4137-80ee-b0f1b36abae7-kube-api-access-lq2fb\" (UniqueName: \"kubernetes.io/projected/a92f9022-bc45-4137-80ee-b0f1b36abae7-kube-api-access-lq2fb\") pod \"a92f9022-bc45-4137-80ee-b0f1b36abae7\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " Oct 30 15:56:42.303487 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.303421 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-xks-ca\") pod \"a92f9022-bc45-4137-80ee-b0f1b36abae7\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " Oct 30 15:56:42.303487 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.303443 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-openshift-ca\") pod \"a92f9022-bc45-4137-80ee-b0f1b36abae7\" (UID: \"a92f9022-bc45-4137-80ee-b0f1b36abae7\") " Oct 30 15:56:42.303881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.303848 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-openshift-ca" pod "a92f9022-bc45-4137-80ee-b0f1b36abae7" (UID: "a92f9022-bc45-4137-80ee-b0f1b36abae7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:42.304062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.304047 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:42.305458 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.305435 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92f9022-bc45-4137-80ee-b0f1b36abae7-kube-api-access-lq2fb" pod "a92f9022-bc45-4137-80ee-b0f1b36abae7" (UID: "a92f9022-bc45-4137-80ee-b0f1b36abae7"). InnerVolumeSpecName "kube-api-access-lq2fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:42.305547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.305501 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-xks-ca" pod "a92f9022-bc45-4137-80ee-b0f1b36abae7" (UID: "a92f9022-bc45-4137-80ee-b0f1b36abae7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:42.405730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.405687 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a92f9022-bc45-4137-80ee-b0f1b36abae7-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:42.405730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:42.405720 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2fb\" (UniqueName: \"kubernetes.io/projected/a92f9022-bc45-4137-80ee-b0f1b36abae7-kube-api-access-lq2fb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:43.010078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.010051 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" Oct 30 15:56:43.010502 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.010089 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k" event={"ID":"a92f9022-bc45-4137-80ee-b0f1b36abae7","Type":"ContainerDied","Data":"ea5f0bd2c184fa39fcde3f6e9d8d6197975184a6ce4c7ff72e8289af3c2a2667"} Oct 30 15:56:43.010502 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.010134 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea5f0bd2c184fa39fcde3f6e9d8d6197975184a6ce4c7ff72e8289af3c2a2667" Oct 30 15:56:43.292938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.292906 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:43.334847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.334788 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-kube-api-access-5w7gb\" (UniqueName: \"kubernetes.io/projected/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-kube-api-access-5w7gb\") pod \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " Oct 30 15:56:43.335050 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.334917 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-xks-ca\") pod \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " Oct 30 15:56:43.335116 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.335064 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-openshift-ca\") pod \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\" (UID: \"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7\") " Oct 30 15:56:43.335417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.335390 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-openshift-ca" pod "f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" (UID: "f4a04bd0-7ab3-450f-9139-3ab5d8993aa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:43.335693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.335675 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:43.337434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.337400 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-kube-api-access-5w7gb" pod "f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" (UID: "f4a04bd0-7ab3-450f-9139-3ab5d8993aa7"). InnerVolumeSpecName "kube-api-access-5w7gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:43.337561 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.337471 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-xks-ca" pod "f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" (UID: "f4a04bd0-7ab3-450f-9139-3ab5d8993aa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:43.437888 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.437855 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7gb\" (UniqueName: \"kubernetes.io/projected/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-kube-api-access-5w7gb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:43.438041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:43.437893 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:44.020785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.020747 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" Oct 30 15:56:44.020785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.020775 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg" event={"ID":"f4a04bd0-7ab3-450f-9139-3ab5d8993aa7","Type":"ContainerDied","Data":"01d81fbeb02010b03241ce7cb09fcd5e19c3413d468df4f10b03e12191b6bd82"} Oct 30 15:56:44.021255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.020850 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01d81fbeb02010b03241ce7cb09fcd5e19c3413d468df4f10b03e12191b6bd82" Oct 30 15:56:44.556055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556020 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf"] Oct 30 15:56:44.556711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556698 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a92f9022-bc45-4137-80ee-b0f1b36abae7" containerName="revoke-keys" Oct 30 15:56:44.556765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556712 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92f9022-bc45-4137-80ee-b0f1b36abae7" containerName="revoke-keys" Oct 30 15:56:44.556765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556727 4401 cpu_manager.go:414] "containerMap: removing container" podUID="f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" containerName="revoke-keys" Oct 30 15:56:44.556765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556732 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" containerName="revoke-keys" Oct 30 15:56:44.556765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556745 4401 cpu_manager.go:414] "containerMap: removing container" podUID="26631d06-17b9-4ce8-80d5-ab4dcc21cf19" containerName="revoke-keys" Oct 30 15:56:44.556765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556750 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26631d06-17b9-4ce8-80d5-ab4dcc21cf19" containerName="revoke-keys" Oct 30 15:56:44.556959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556916 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92f9022-bc45-4137-80ee-b0f1b36abae7" containerName="revoke-keys" Oct 30 15:56:44.556959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556926 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" containerName="revoke-keys" Oct 30 15:56:44.556959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.556933 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="26631d06-17b9-4ce8-80d5-ab4dcc21cf19" containerName="revoke-keys" Oct 30 15:56:44.560890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.560868 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.577675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.577629 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:56:44.580376 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.580333 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf"] Oct 30 15:56:44.586274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.586238 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:56:44.586488 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.586471 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:56:44.586566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.586550 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:56:44.656960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.656922 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.657153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.657076 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lxb\" (UniqueName: \"kubernetes.io/projected/d07ba13d-aa01-4829-a390-2f3378833212-kube-api-access-95lxb\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.657231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.657218 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.758710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.758668 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.758938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.758757 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lxb\" (UniqueName: \"kubernetes.io/projected/d07ba13d-aa01-4829-a390-2f3378833212-kube-api-access-95lxb\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.758938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.758867 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.759402 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.759380 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.761306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.761280 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.785310 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.785280 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lxb\" (UniqueName: \"kubernetes.io/projected/d07ba13d-aa01-4829-a390-2f3378833212-kube-api-access-95lxb\") pod \"maas-api-revoke-sub-efc30c9e-vjxxf\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:44.890863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:44.887465 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:45.270786 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:45.270755 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07ba13d_aa01_4829_a390_2f3378833212.slice/crio-59a941c7e3d2f73c7a8faf9ac75811bb294576c84131aa40dd2fe3aa6b6dafbd WatchSource:0}: Error finding container 59a941c7e3d2f73c7a8faf9ac75811bb294576c84131aa40dd2fe3aa6b6dafbd: Status 404 returned error can't find the container with id 59a941c7e3d2f73c7a8faf9ac75811bb294576c84131aa40dd2fe3aa6b6dafbd Oct 30 15:56:45.273128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:45.273103 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf"] Oct 30 15:56:46.043551 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:46.043509 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" event={"ID":"d07ba13d-aa01-4829-a390-2f3378833212","Type":"ContainerStarted","Data":"0d214dab73ec886783f02cdb688a34969c8dcdae126a7071f72335827e771a34"} Oct 30 15:56:46.043752 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:46.043576 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" event={"ID":"d07ba13d-aa01-4829-a390-2f3378833212","Type":"ContainerStarted","Data":"59a941c7e3d2f73c7a8faf9ac75811bb294576c84131aa40dd2fe3aa6b6dafbd"} Oct 30 15:56:46.088939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:46.088877 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" podStartSLOduration=2.088856207 podStartE2EDuration="2.088856207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:46.08694773 +0000 UTC m=+1114.786220886" watchObservedRunningTime="2026-10-30 15:56:46.088856207 +0000 UTC m=+1114.788129362" Oct 30 15:56:47.055315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:47.055277 4401 generic.go:356] "Generic (PLEG): container finished" podID="d07ba13d-aa01-4829-a390-2f3378833212" containerID="0d214dab73ec886783f02cdb688a34969c8dcdae126a7071f72335827e771a34" exitCode=0 Oct 30 15:56:47.055742 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:47.055326 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" event={"ID":"d07ba13d-aa01-4829-a390-2f3378833212","Type":"ContainerDied","Data":"0d214dab73ec886783f02cdb688a34969c8dcdae126a7071f72335827e771a34"} Oct 30 15:56:48.241089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.241052 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:48.319919 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.319799 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-openshift-ca\") pod \"d07ba13d-aa01-4829-a390-2f3378833212\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " Oct 30 15:56:48.320096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.320066 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07ba13d-aa01-4829-a390-2f3378833212-kube-api-access-95lxb\" (UniqueName: \"kubernetes.io/projected/d07ba13d-aa01-4829-a390-2f3378833212-kube-api-access-95lxb\") pod \"d07ba13d-aa01-4829-a390-2f3378833212\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " Oct 30 15:56:48.320158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.320119 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-xks-ca\") pod \"d07ba13d-aa01-4829-a390-2f3378833212\" (UID: \"d07ba13d-aa01-4829-a390-2f3378833212\") " Oct 30 15:56:48.320296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.320262 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-openshift-ca" pod "d07ba13d-aa01-4829-a390-2f3378833212" (UID: "d07ba13d-aa01-4829-a390-2f3378833212"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:48.320757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.320732 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:48.323171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.323131 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-xks-ca" pod "d07ba13d-aa01-4829-a390-2f3378833212" (UID: "d07ba13d-aa01-4829-a390-2f3378833212"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:48.327082 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.327039 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07ba13d-aa01-4829-a390-2f3378833212-kube-api-access-95lxb" pod "d07ba13d-aa01-4829-a390-2f3378833212" (UID: "d07ba13d-aa01-4829-a390-2f3378833212"). InnerVolumeSpecName "kube-api-access-95lxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:48.423282 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.423132 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lxb\" (UniqueName: \"kubernetes.io/projected/d07ba13d-aa01-4829-a390-2f3378833212-kube-api-access-95lxb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:48.423282 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:48.423173 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07ba13d-aa01-4829-a390-2f3378833212-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:49.076142 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.076105 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" event={"ID":"d07ba13d-aa01-4829-a390-2f3378833212","Type":"ContainerDied","Data":"59a941c7e3d2f73c7a8faf9ac75811bb294576c84131aa40dd2fe3aa6b6dafbd"} Oct 30 15:56:49.076325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.076146 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf" Oct 30 15:56:49.076325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.076157 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59a941c7e3d2f73c7a8faf9ac75811bb294576c84131aa40dd2fe3aa6b6dafbd" Oct 30 15:56:49.649112 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.649062 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92"] Oct 30 15:56:49.650010 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.649983 4401 cpu_manager.go:414] "containerMap: removing container" podUID="d07ba13d-aa01-4829-a390-2f3378833212" containerName="revoke-keys" Oct 30 15:56:49.650010 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.650009 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07ba13d-aa01-4829-a390-2f3378833212" containerName="revoke-keys" Oct 30 15:56:49.650227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.650212 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07ba13d-aa01-4829-a390-2f3378833212" containerName="revoke-keys" Oct 30 15:56:49.654319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.654295 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.671927 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.671793 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92"] Oct 30 15:56:49.678418 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.678376 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:56:49.678418 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.678413 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:56:49.678639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.678472 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:56:49.686455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.686412 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:56:49.742589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.742544 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.742872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.742806 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/3afb2fdc-253a-4459-bac4-07d0a00faba4-kube-api-access-h6xd6\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.742950 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.742935 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.844231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.844188 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/3afb2fdc-253a-4459-bac4-07d0a00faba4-kube-api-access-h6xd6\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.844444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.844269 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.844444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.844347 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.845230 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.845198 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.847021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.846994 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.870193 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.870156 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/3afb2fdc-253a-4459-bac4-07d0a00faba4-kube-api-access-h6xd6\") pod \"maas-api-revoke-sub-040e4fa2-56t92\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:49.971909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:49.971804 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:50.367268 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:50.367236 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afb2fdc_253a_4459_bac4_07d0a00faba4.slice/crio-5fb8183b1eb8749e7e073b8003aa116bcd5e3e20ce3b1618075afea4c4312d0e WatchSource:0}: Error finding container 5fb8183b1eb8749e7e073b8003aa116bcd5e3e20ce3b1618075afea4c4312d0e: Status 404 returned error can't find the container with id 5fb8183b1eb8749e7e073b8003aa116bcd5e3e20ce3b1618075afea4c4312d0e Oct 30 15:56:50.369201 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:50.369175 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92"] Oct 30 15:56:51.224624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:51.224585 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" event={"ID":"3afb2fdc-253a-4459-bac4-07d0a00faba4","Type":"ContainerStarted","Data":"6711f08f4c4043ef2804e28e49149d68965adc52516925c2fe611643fc7a91bd"} Oct 30 15:56:51.225145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:51.224650 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" event={"ID":"3afb2fdc-253a-4459-bac4-07d0a00faba4","Type":"ContainerStarted","Data":"5fb8183b1eb8749e7e073b8003aa116bcd5e3e20ce3b1618075afea4c4312d0e"} Oct 30 15:56:51.273859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:51.273770 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" podStartSLOduration=2.273754711 podStartE2EDuration="2.273754711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:51.270969673 +0000 UTC m=+1119.970242829" watchObservedRunningTime="2026-10-30 15:56:51.273754711 +0000 UTC m=+1119.973027867" Oct 30 15:56:52.020979 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:56:52.020930 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:56:52.236285 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.236247 4401 generic.go:356] "Generic (PLEG): container finished" podID="3afb2fdc-253a-4459-bac4-07d0a00faba4" containerID="6711f08f4c4043ef2804e28e49149d68965adc52516925c2fe611643fc7a91bd" exitCode=0 Oct 30 15:56:52.236673 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.236321 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" event={"ID":"3afb2fdc-253a-4459-bac4-07d0a00faba4","Type":"ContainerDied","Data":"6711f08f4c4043ef2804e28e49149d68965adc52516925c2fe611643fc7a91bd"} Oct 30 15:56:52.870125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.870083 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc"] Oct 30 15:56:52.876312 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.876284 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:52.894906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.894801 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc"] Oct 30 15:56:52.973353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.973314 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96jz\" (UniqueName: \"kubernetes.io/projected/5e143195-565b-4ce3-8638-c7ba6b23ab27-kube-api-access-r96jz\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:52.973505 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.973486 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:52.973617 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:52.973598 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.079306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.077654 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96jz\" (UniqueName: \"kubernetes.io/projected/5e143195-565b-4ce3-8638-c7ba6b23ab27-kube-api-access-r96jz\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.079625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.079595 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.079775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.079756 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.080348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.080322 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.082243 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.082213 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.103100 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.103068 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96jz\" (UniqueName: \"kubernetes.io/projected/5e143195-565b-4ce3-8638-c7ba6b23ab27-kube-api-access-r96jz\") pod \"maas-api-revoke-sub-68d04e69-ts5lc\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.200724 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.200635 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:53.530309 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.530250 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:53.559210 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:56:53.559172 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e143195_565b_4ce3_8638_c7ba6b23ab27.slice/crio-b6e0e1b44e0cfaed1a3f76532c5407d39e763dec4c38b75e2c93ac60cb78b755 WatchSource:0}: Error finding container b6e0e1b44e0cfaed1a3f76532c5407d39e763dec4c38b75e2c93ac60cb78b755: Status 404 returned error can't find the container with id b6e0e1b44e0cfaed1a3f76532c5407d39e763dec4c38b75e2c93ac60cb78b755 Oct 30 15:56:53.568085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.568054 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc"] Oct 30 15:56:53.692636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.692603 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-xks-ca\") pod \"3afb2fdc-253a-4459-bac4-07d0a00faba4\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " Oct 30 15:56:53.692815 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.692722 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-openshift-ca\") pod \"3afb2fdc-253a-4459-bac4-07d0a00faba4\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " Oct 30 15:56:53.692919 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.692877 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afb2fdc-253a-4459-bac4-07d0a00faba4-kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/3afb2fdc-253a-4459-bac4-07d0a00faba4-kube-api-access-h6xd6\") pod \"3afb2fdc-253a-4459-bac4-07d0a00faba4\" (UID: \"3afb2fdc-253a-4459-bac4-07d0a00faba4\") " Oct 30 15:56:53.693109 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.693079 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-openshift-ca" pod "3afb2fdc-253a-4459-bac4-07d0a00faba4" (UID: "3afb2fdc-253a-4459-bac4-07d0a00faba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:53.693410 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.693394 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:53.747676 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.747635 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afb2fdc-253a-4459-bac4-07d0a00faba4-kube-api-access-h6xd6" pod "3afb2fdc-253a-4459-bac4-07d0a00faba4" (UID: "3afb2fdc-253a-4459-bac4-07d0a00faba4"). InnerVolumeSpecName "kube-api-access-h6xd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:53.747885 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.747709 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-xks-ca" pod "3afb2fdc-253a-4459-bac4-07d0a00faba4" (UID: "3afb2fdc-253a-4459-bac4-07d0a00faba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:53.805986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.803495 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6xd6\" (UniqueName: \"kubernetes.io/projected/3afb2fdc-253a-4459-bac4-07d0a00faba4-kube-api-access-h6xd6\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:53.805986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:53.803538 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afb2fdc-253a-4459-bac4-07d0a00faba4-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:54.275793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:54.275700 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" event={"ID":"5e143195-565b-4ce3-8638-c7ba6b23ab27","Type":"ContainerStarted","Data":"ff64d5ae884cae0a7f685b8f347529912b9c9ca5939a4e679d98e07822ae438a"} Oct 30 15:56:54.275793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:54.275763 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" event={"ID":"5e143195-565b-4ce3-8638-c7ba6b23ab27","Type":"ContainerStarted","Data":"b6e0e1b44e0cfaed1a3f76532c5407d39e763dec4c38b75e2c93ac60cb78b755"} Oct 30 15:56:54.277799 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:54.277772 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" Oct 30 15:56:54.277971 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:54.277791 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92" event={"ID":"3afb2fdc-253a-4459-bac4-07d0a00faba4","Type":"ContainerDied","Data":"5fb8183b1eb8749e7e073b8003aa116bcd5e3e20ce3b1618075afea4c4312d0e"} Oct 30 15:56:54.277971 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:54.277854 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fb8183b1eb8749e7e073b8003aa116bcd5e3e20ce3b1618075afea4c4312d0e" Oct 30 15:56:54.319757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:54.319632 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" podStartSLOduration=2.319617349 podStartE2EDuration="2.319617349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:56:54.319156064 +0000 UTC m=+1123.018429221" watchObservedRunningTime="2026-10-30 15:56:54.319617349 +0000 UTC m=+1123.018890506" Oct 30 15:56:55.293335 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:55.293285 4401 generic.go:356] "Generic (PLEG): container finished" podID="5e143195-565b-4ce3-8638-c7ba6b23ab27" containerID="ff64d5ae884cae0a7f685b8f347529912b9c9ca5939a4e679d98e07822ae438a" exitCode=0 Oct 30 15:56:55.293948 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:55.293348 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" event={"ID":"5e143195-565b-4ce3-8638-c7ba6b23ab27","Type":"ContainerDied","Data":"ff64d5ae884cae0a7f685b8f347529912b9c9ca5939a4e679d98e07822ae438a"} Oct 30 15:56:55.513398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:55.513361 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:56:57.334105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.334066 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:57.380906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.373391 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-xks-ca\") pod \"5e143195-565b-4ce3-8638-c7ba6b23ab27\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " Oct 30 15:56:57.380906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.374090 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e143195-565b-4ce3-8638-c7ba6b23ab27-kube-api-access-r96jz\" (UniqueName: \"kubernetes.io/projected/5e143195-565b-4ce3-8638-c7ba6b23ab27-kube-api-access-r96jz\") pod \"5e143195-565b-4ce3-8638-c7ba6b23ab27\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " Oct 30 15:56:57.380906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.374563 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-openshift-ca\") pod \"5e143195-565b-4ce3-8638-c7ba6b23ab27\" (UID: \"5e143195-565b-4ce3-8638-c7ba6b23ab27\") " Oct 30 15:56:57.695119 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.679301 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e143195-565b-4ce3-8638-c7ba6b23ab27-kube-api-access-r96jz" pod "5e143195-565b-4ce3-8638-c7ba6b23ab27" (UID: "5e143195-565b-4ce3-8638-c7ba6b23ab27"). InnerVolumeSpecName "kube-api-access-r96jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:56:57.729351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.697532 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" Oct 30 15:56:57.729351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.704164 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96jz\" (UniqueName: \"kubernetes.io/projected/5e143195-565b-4ce3-8638-c7ba6b23ab27-kube-api-access-r96jz\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:57.729351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.704453 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc" event={"ID":"5e143195-565b-4ce3-8638-c7ba6b23ab27","Type":"ContainerDied","Data":"b6e0e1b44e0cfaed1a3f76532c5407d39e763dec4c38b75e2c93ac60cb78b755"} Oct 30 15:56:57.729351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.704513 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e0e1b44e0cfaed1a3f76532c5407d39e763dec4c38b75e2c93ac60cb78b755" Oct 30 15:56:57.729351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.724527 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-openshift-ca" pod "5e143195-565b-4ce3-8638-c7ba6b23ab27" (UID: "5e143195-565b-4ce3-8638-c7ba6b23ab27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:56:57.753873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.743483 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-xks-ca" pod "5e143195-565b-4ce3-8638-c7ba6b23ab27" (UID: "5e143195-565b-4ce3-8638-c7ba6b23ab27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:56:57.820854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.820350 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:57.820854 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.820393 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e143195-565b-4ce3-8638-c7ba6b23ab27-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:56:57.971913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:57.970975 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:56:58.015877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.009937 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:58.229905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.227928 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:58.229905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.228357 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:56:58.448191 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.424858 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:56:58.448191 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.425253 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:56:58.516912 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.496800 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:58.854839 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.854750 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:58.980105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.980044 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:59.003853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:58.981372 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:56:59.074911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.031774 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:56:59.096863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.075971 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:56:59.185913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.178345 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:59.265304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.265234 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:59.266365 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.266314 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:56:59.295177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.295140 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:56:59.295521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.295500 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:56:59.437914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.425531 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:56:59.437914 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.425956 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:56:59.523857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:56:59.497061 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:00.021889 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.008716 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:57:00.115852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.086745 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:00.142735 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.105401 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:57:00.143059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.143038 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:57:00.160908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.152009 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:57:00.295843 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.270071 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:00.312213 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.312058 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:00.508851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.494221 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:57:00.509158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.496385 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:00.512848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.496619 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:00.531916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:00.498462 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:01.015639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.015593 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:01.154966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.154911 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:57:01.329076 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.328967 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:57:01.377099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.377048 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:01.531787 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.531737 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:01.534149 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.532755 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:57:01.549116 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.549055 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:57:01.550296 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.550249 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:57:01.550628 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.550579 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:01.888878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:01.871678 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:02.020855 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.020614 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:57:02.031118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.031084 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:57:02.282903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.275306 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:57:02.282903 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.278994 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:02.340154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.340092 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:02.384014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.383963 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:57:02.532378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.532322 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:57:02.614848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:02.607589 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:03.299848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:03.297210 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:57:03.428449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:03.428397 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:03.686140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:03.677253 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:57:03.816238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:03.816185 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:04.102911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.077588 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:04.208926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.191354 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:57:04.306855 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.302124 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:04.503913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.432162 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:57:04.546026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.545992 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:57:04.618313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.618280 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:57:04.618895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.618872 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:57:04.619081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.619068 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:57:04.761426 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.761331 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp"] Oct 30 15:57:04.765935 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.765893 4401 cpu_manager.go:414] "containerMap: removing container" podUID="3afb2fdc-253a-4459-bac4-07d0a00faba4" containerName="revoke-keys" Oct 30 15:57:04.766150 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.766133 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afb2fdc-253a-4459-bac4-07d0a00faba4" containerName="revoke-keys" Oct 30 15:57:04.766385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.766366 4401 cpu_manager.go:414] "containerMap: removing container" podUID="5e143195-565b-4ce3-8638-c7ba6b23ab27" containerName="revoke-keys" Oct 30 15:57:04.766490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.766478 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e143195-565b-4ce3-8638-c7ba6b23ab27" containerName="revoke-keys" Oct 30 15:57:04.766834 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.766806 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e143195-565b-4ce3-8638-c7ba6b23ab27" containerName="revoke-keys" Oct 30 15:57:04.766940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.766928 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afb2fdc-253a-4459-bac4-07d0a00faba4" containerName="revoke-keys" Oct 30 15:57:04.773938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.773896 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp"] Oct 30 15:57:04.774114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.774015 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:04.810919 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.810882 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs"] Oct 30 15:57:04.813363 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.813327 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9e2b29\"/\"distinct-w2-9e2b29-kserve-self-signed-certs\"" Oct 30 15:57:04.822065 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.822033 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs"] Oct 30 15:57:04.834332 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.824353 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:04.848356 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.848331 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:57:04.849765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.849738 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:57:04.896382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.896349 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:57:04.896588 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.896486 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.950913 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-model-cache\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.950978 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-tmp-dir\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.951030 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.951066 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f10fad-1159-4cc9-88b7-9da9159371a1-tls-certs\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.951119 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkx9\" (UniqueName: \"kubernetes.io/projected/b1f10fad-1159-4cc9-88b7-9da9159371a1-kube-api-access-blkx9\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.951158 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-dshm\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.951188 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-home\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.951346 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkshg\" (UniqueName: \"kubernetes.io/projected/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-kube-api-access-gkshg\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:04.951746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:04.951488 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.095902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.095679 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-model-cache\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.095902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.095732 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-tmp-dir\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.095902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.095781 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.096223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.095817 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f10fad-1159-4cc9-88b7-9da9159371a1-tls-certs\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.096223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.096023 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkx9\" (UniqueName: \"kubernetes.io/projected/b1f10fad-1159-4cc9-88b7-9da9159371a1-kube-api-access-blkx9\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.096223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.096095 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-dshm\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.096223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.096131 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-home\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.096419 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.096286 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkshg\" (UniqueName: \"kubernetes.io/projected/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-kube-api-access-gkshg\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.096419 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.096381 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-tmp-dir\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.096521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.096433 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.103848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.097337 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.105020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.104166 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-home\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.105641 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.105611 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-model-cache\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.107817 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.106267 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-dshm\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.118679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.118502 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f10fad-1159-4cc9-88b7-9da9159371a1-tls-certs\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.134231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.134193 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.151009 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.150473 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkshg\" (UniqueName: \"kubernetes.io/projected/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-kube-api-access-gkshg\") pod \"maas-api-revoke-sub-05a14e06-l8gbs\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.151540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.151503 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkx9\" (UniqueName: \"kubernetes.io/projected/b1f10fad-1159-4cc9-88b7-9da9159371a1-kube-api-access-blkx9\") pod \"distinct-w2-9e2b29-kserve-b66bb88-48xhp\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.151757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.151739 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:05.443909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.439159 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:05.544868 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.540841 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs"] Oct 30 15:57:05.549583 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:05.548523 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc258d9_c4f1_40c5_ab67_0fe3d613698f.slice/crio-75c1edd2b797d2f640f066642f6b46c71d8cd86ba2af6533c9f8a8c186866b8c WatchSource:0}: Error finding container 75c1edd2b797d2f640f066642f6b46c71d8cd86ba2af6533c9f8a8c186866b8c: Status 404 returned error can't find the container with id 75c1edd2b797d2f640f066642f6b46c71d8cd86ba2af6533c9f8a8c186866b8c Oct 30 15:57:05.625040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.621942 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" event={"ID":"cfc258d9-c4f1-40c5-ab67-0fe3d613698f","Type":"ContainerStarted","Data":"75c1edd2b797d2f640f066642f6b46c71d8cd86ba2af6533c9f8a8c186866b8c"} Oct 30 15:57:05.738641 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.738542 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch"] Oct 30 15:57:05.746185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.746157 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:05.762632 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.762595 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch"] Oct 30 15:57:05.776582 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:05.776547 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f10fad_1159_4cc9_88b7_9da9159371a1.slice/crio-f2cbdb966918bb8a0e78142bb98a860c231644b176a0e8b1ef07bb077a1918bb WatchSource:0}: Error finding container f2cbdb966918bb8a0e78142bb98a860c231644b176a0e8b1ef07bb077a1918bb: Status 404 returned error can't find the container with id f2cbdb966918bb8a0e78142bb98a860c231644b176a0e8b1ef07bb077a1918bb Oct 30 15:57:05.794899 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.787922 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp"] Oct 30 15:57:05.916030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.915985 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:05.916214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.916091 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:05.916372 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:05.916351 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4frh\" (UniqueName: \"kubernetes.io/projected/f5e23559-203e-42f8-84a6-8e9bb10ad432-kube-api-access-q4frh\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.017554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.017459 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4frh\" (UniqueName: \"kubernetes.io/projected/f5e23559-203e-42f8-84a6-8e9bb10ad432-kube-api-access-q4frh\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.017716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.017552 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.017716 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.017620 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.018359 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.018330 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.046576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.046535 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.046810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.046788 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4frh\" (UniqueName: \"kubernetes.io/projected/f5e23559-203e-42f8-84a6-8e9bb10ad432-kube-api-access-q4frh\") pod \"maas-api-revoke-sub-13c91299-m79ch\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.065703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.065671 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:06.381968 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:06.381934 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e23559_203e_42f8_84a6_8e9bb10ad432.slice/crio-990cf4a727606d7e54415bf3e4626af4cb5798409e3fac79dd14cc080972d504 WatchSource:0}: Error finding container 990cf4a727606d7e54415bf3e4626af4cb5798409e3fac79dd14cc080972d504: Status 404 returned error can't find the container with id 990cf4a727606d7e54415bf3e4626af4cb5798409e3fac79dd14cc080972d504 Oct 30 15:57:06.384336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.384312 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch"] Oct 30 15:57:06.633432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.633327 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" event={"ID":"cfc258d9-c4f1-40c5-ab67-0fe3d613698f","Type":"ContainerStarted","Data":"058e602cf112d62b3d6883d30a0ede8b6a871b036a59a6c783cfa6e395516e1b"} Oct 30 15:57:06.635271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.635238 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" event={"ID":"b1f10fad-1159-4cc9-88b7-9da9159371a1","Type":"ContainerStarted","Data":"eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b"} Oct 30 15:57:06.635429 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.635282 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" event={"ID":"b1f10fad-1159-4cc9-88b7-9da9159371a1","Type":"ContainerStarted","Data":"f2cbdb966918bb8a0e78142bb98a860c231644b176a0e8b1ef07bb077a1918bb"} Oct 30 15:57:06.636747 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.636721 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" event={"ID":"f5e23559-203e-42f8-84a6-8e9bb10ad432","Type":"ContainerStarted","Data":"990cf4a727606d7e54415bf3e4626af4cb5798409e3fac79dd14cc080972d504"} Oct 30 15:57:06.685001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.684947 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" podStartSLOduration=3.684932118 podStartE2EDuration="3.684932118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:06.679297799 +0000 UTC m=+1135.378570956" watchObservedRunningTime="2026-10-30 15:57:06.684932118 +0000 UTC m=+1135.384205275" Oct 30 15:57:06.734925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:06.734867 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podStartSLOduration=3.734850215 podStartE2EDuration="3.734850215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:06.728777789 +0000 UTC m=+1135.428050947" watchObservedRunningTime="2026-10-30 15:57:06.734850215 +0000 UTC m=+1135.434123369" Oct 30 15:57:07.444146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:07.444110 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:07.648188 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:07.648094 4401 generic.go:356] "Generic (PLEG): container finished" podID="f5e23559-203e-42f8-84a6-8e9bb10ad432" containerID="5e99be56efb0aced5f07116f2f946b58e549da69a78ee3c6be84e0e493650452" exitCode=0 Oct 30 15:57:07.648351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:07.648175 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" event={"ID":"f5e23559-203e-42f8-84a6-8e9bb10ad432","Type":"ContainerDied","Data":"5e99be56efb0aced5f07116f2f946b58e549da69a78ee3c6be84e0e493650452"} Oct 30 15:57:07.650071 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:07.650043 4401 generic.go:356] "Generic (PLEG): container finished" podID="cfc258d9-c4f1-40c5-ab67-0fe3d613698f" containerID="058e602cf112d62b3d6883d30a0ede8b6a871b036a59a6c783cfa6e395516e1b" exitCode=0 Oct 30 15:57:07.650205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:07.650124 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" event={"ID":"cfc258d9-c4f1-40c5-ab67-0fe3d613698f","Type":"ContainerDied","Data":"058e602cf112d62b3d6883d30a0ede8b6a871b036a59a6c783cfa6e395516e1b"} Oct 30 15:57:08.885242 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.885213 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:08.889148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.889078 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:08.971911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.971757 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-kube-api-access-gkshg\" (UniqueName: \"kubernetes.io/projected/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-kube-api-access-gkshg\") pod \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " Oct 30 15:57:08.971911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.971855 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-openshift-ca\") pod \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " Oct 30 15:57:08.972199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.972171 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5e23559-203e-42f8-84a6-8e9bb10ad432-kube-api-access-q4frh\" (UniqueName: \"kubernetes.io/projected/f5e23559-203e-42f8-84a6-8e9bb10ad432-kube-api-access-q4frh\") pod \"f5e23559-203e-42f8-84a6-8e9bb10ad432\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " Oct 30 15:57:08.972336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.972220 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-openshift-ca\") pod \"f5e23559-203e-42f8-84a6-8e9bb10ad432\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " Oct 30 15:57:08.972336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.972262 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-xks-ca\") pod \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\" (UID: \"cfc258d9-c4f1-40c5-ab67-0fe3d613698f\") " Oct 30 15:57:08.972336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.972323 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-xks-ca\") pod \"f5e23559-203e-42f8-84a6-8e9bb10ad432\" (UID: \"f5e23559-203e-42f8-84a6-8e9bb10ad432\") " Oct 30 15:57:08.972517 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.972456 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-openshift-ca" pod "cfc258d9-c4f1-40c5-ab67-0fe3d613698f" (UID: "cfc258d9-c4f1-40c5-ab67-0fe3d613698f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:08.972696 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.972664 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-openshift-ca" pod "f5e23559-203e-42f8-84a6-8e9bb10ad432" (UID: "f5e23559-203e-42f8-84a6-8e9bb10ad432"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:08.973120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.973052 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:08.973120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.973082 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:08.974796 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.974760 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-xks-ca" pod "cfc258d9-c4f1-40c5-ab67-0fe3d613698f" (UID: "cfc258d9-c4f1-40c5-ab67-0fe3d613698f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:08.974997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.974962 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e23559-203e-42f8-84a6-8e9bb10ad432-kube-api-access-q4frh" pod "f5e23559-203e-42f8-84a6-8e9bb10ad432" (UID: "f5e23559-203e-42f8-84a6-8e9bb10ad432"). InnerVolumeSpecName "kube-api-access-q4frh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:08.975599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.975572 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-xks-ca" pod "f5e23559-203e-42f8-84a6-8e9bb10ad432" (UID: "f5e23559-203e-42f8-84a6-8e9bb10ad432"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:08.975772 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:08.975753 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-kube-api-access-gkshg" pod "cfc258d9-c4f1-40c5-ab67-0fe3d613698f" (UID: "cfc258d9-c4f1-40c5-ab67-0fe3d613698f"). InnerVolumeSpecName "kube-api-access-gkshg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:09.075074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.075034 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4frh\" (UniqueName: \"kubernetes.io/projected/f5e23559-203e-42f8-84a6-8e9bb10ad432-kube-api-access-q4frh\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:09.075074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.075069 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:09.075074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.075081 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5e23559-203e-42f8-84a6-8e9bb10ad432-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:09.075313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.075089 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkshg\" (UniqueName: \"kubernetes.io/projected/cfc258d9-c4f1-40c5-ab67-0fe3d613698f-kube-api-access-gkshg\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:09.678178 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.678146 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" Oct 30 15:57:09.678178 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.678160 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch" event={"ID":"f5e23559-203e-42f8-84a6-8e9bb10ad432","Type":"ContainerDied","Data":"990cf4a727606d7e54415bf3e4626af4cb5798409e3fac79dd14cc080972d504"} Oct 30 15:57:09.678431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.678220 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="990cf4a727606d7e54415bf3e4626af4cb5798409e3fac79dd14cc080972d504" Oct 30 15:57:09.680156 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.680129 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" Oct 30 15:57:09.680297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.680162 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs" event={"ID":"cfc258d9-c4f1-40c5-ab67-0fe3d613698f","Type":"ContainerDied","Data":"75c1edd2b797d2f640f066642f6b46c71d8cd86ba2af6533c9f8a8c186866b8c"} Oct 30 15:57:09.680297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:09.680212 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c1edd2b797d2f640f066642f6b46c71d8cd86ba2af6533c9f8a8c186866b8c" Oct 30 15:57:14.092565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.092528 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl"] Oct 30 15:57:14.094722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.094689 4401 cpu_manager.go:414] "containerMap: removing container" podUID="f5e23559-203e-42f8-84a6-8e9bb10ad432" containerName="revoke-keys" Oct 30 15:57:14.094953 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.094936 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5e23559-203e-42f8-84a6-8e9bb10ad432" containerName="revoke-keys" Oct 30 15:57:14.095147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.095135 4401 cpu_manager.go:414] "containerMap: removing container" podUID="cfc258d9-c4f1-40c5-ab67-0fe3d613698f" containerName="revoke-keys" Oct 30 15:57:14.095251 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.095240 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc258d9-c4f1-40c5-ab67-0fe3d613698f" containerName="revoke-keys" Oct 30 15:57:14.105871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.102077 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5e23559-203e-42f8-84a6-8e9bb10ad432" containerName="revoke-keys" Oct 30 15:57:14.105871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.102135 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc258d9-c4f1-40c5-ab67-0fe3d613698f" containerName="revoke-keys" Oct 30 15:57:14.112196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.110862 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.116411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.116373 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl"] Oct 30 15:57:14.131087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.131047 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:57:14.131087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.131060 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:57:14.131329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.131158 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:57:14.131329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.131188 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:57:14.172208 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.172165 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.172403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.172290 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq25b\" (UniqueName: \"kubernetes.io/projected/9aafde56-f3e7-415b-83b7-98c52e390add-kube-api-access-xq25b\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.172450 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.172428 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.273986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.273937 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.274199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.274061 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq25b\" (UniqueName: \"kubernetes.io/projected/9aafde56-f3e7-415b-83b7-98c52e390add-kube-api-access-xq25b\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.274331 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.274206 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.274943 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.274918 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.276637 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.276613 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.300055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.300026 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq25b\" (UniqueName: \"kubernetes.io/projected/9aafde56-f3e7-415b-83b7-98c52e390add-kube-api-access-xq25b\") pod \"maas-api-revoke-sub-a59c045e-hzkzl\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.430213 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.430173 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:14.750345 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:14.750312 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aafde56_f3e7_415b_83b7_98c52e390add.slice/crio-8367910d08572894ae835abe9cceec86713672daa7fd4643b871dbe88b57a0ff WatchSource:0}: Error finding container 8367910d08572894ae835abe9cceec86713672daa7fd4643b871dbe88b57a0ff: Status 404 returned error can't find the container with id 8367910d08572894ae835abe9cceec86713672daa7fd4643b871dbe88b57a0ff Oct 30 15:57:14.752485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:14.752447 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl"] Oct 30 15:57:15.444092 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:15.444008 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:15.457759 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:15.457729 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:15.743881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:15.743846 4401 generic.go:356] "Generic (PLEG): container finished" podID="9aafde56-f3e7-415b-83b7-98c52e390add" containerID="34f9e593e3e7daa702a16320c63863c487bca266cd5662541a9694f91492b22c" exitCode=0 Oct 30 15:57:15.744045 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:15.743918 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" event={"ID":"9aafde56-f3e7-415b-83b7-98c52e390add","Type":"ContainerDied","Data":"34f9e593e3e7daa702a16320c63863c487bca266cd5662541a9694f91492b22c"} Oct 30 15:57:15.744045 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:15.743993 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" event={"ID":"9aafde56-f3e7-415b-83b7-98c52e390add","Type":"ContainerStarted","Data":"8367910d08572894ae835abe9cceec86713672daa7fd4643b871dbe88b57a0ff"} Oct 30 15:57:15.756228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:15.756198 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:16.144203 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.144164 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm"] Oct 30 15:57:16.152202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.152159 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.171033 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.170999 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm"] Oct 30 15:57:16.209774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.209717 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.210009 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.209898 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.210009 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.209966 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbl8\" (UniqueName: \"kubernetes.io/projected/faca6252-0610-4f50-aed9-bd07ef8c7938-kube-api-access-xpbl8\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.311132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.311094 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.311330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.311210 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.311330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.311284 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbl8\" (UniqueName: \"kubernetes.io/projected/faca6252-0610-4f50-aed9-bd07ef8c7938-kube-api-access-xpbl8\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.311862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.311814 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.313619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.313596 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.337125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.337087 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbl8\" (UniqueName: \"kubernetes.io/projected/faca6252-0610-4f50-aed9-bd07ef8c7938-kube-api-access-xpbl8\") pod \"maas-api-revoke-sub-4fd6546f-qthjm\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.475037 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.474946 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:16.786603 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:16.786563 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaca6252_0610_4f50_aed9_bd07ef8c7938.slice/crio-32ea1becb12d1a177b304a92abd9e122fb5800e8f41042bce5506261f9c2f601 WatchSource:0}: Error finding container 32ea1becb12d1a177b304a92abd9e122fb5800e8f41042bce5506261f9c2f601: Status 404 returned error can't find the container with id 32ea1becb12d1a177b304a92abd9e122fb5800e8f41042bce5506261f9c2f601 Oct 30 15:57:16.791980 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.791950 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm"] Oct 30 15:57:16.925177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:16.925143 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:17.021131 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.021048 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aafde56-f3e7-415b-83b7-98c52e390add-kube-api-access-xq25b\" (UniqueName: \"kubernetes.io/projected/9aafde56-f3e7-415b-83b7-98c52e390add-kube-api-access-xq25b\") pod \"9aafde56-f3e7-415b-83b7-98c52e390add\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " Oct 30 15:57:17.021286 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.021136 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-xks-ca\") pod \"9aafde56-f3e7-415b-83b7-98c52e390add\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " Oct 30 15:57:17.021286 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.021158 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-openshift-ca\") pod \"9aafde56-f3e7-415b-83b7-98c52e390add\" (UID: \"9aafde56-f3e7-415b-83b7-98c52e390add\") " Oct 30 15:57:17.021648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.021618 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-openshift-ca" pod "9aafde56-f3e7-415b-83b7-98c52e390add" (UID: "9aafde56-f3e7-415b-83b7-98c52e390add"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:17.021787 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.021767 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:17.023383 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.023353 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-xks-ca" pod "9aafde56-f3e7-415b-83b7-98c52e390add" (UID: "9aafde56-f3e7-415b-83b7-98c52e390add"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:17.023474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.023365 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aafde56-f3e7-415b-83b7-98c52e390add-kube-api-access-xq25b" pod "9aafde56-f3e7-415b-83b7-98c52e390add" (UID: "9aafde56-f3e7-415b-83b7-98c52e390add"). InnerVolumeSpecName "kube-api-access-xq25b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:17.123390 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.123339 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq25b\" (UniqueName: \"kubernetes.io/projected/9aafde56-f3e7-415b-83b7-98c52e390add-kube-api-access-xq25b\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:17.123390 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.123382 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aafde56-f3e7-415b-83b7-98c52e390add-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:17.399523 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.399480 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq"] Oct 30 15:57:17.400275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.400255 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9aafde56-f3e7-415b-83b7-98c52e390add" containerName="revoke-keys" Oct 30 15:57:17.400275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.400272 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aafde56-f3e7-415b-83b7-98c52e390add" containerName="revoke-keys" Oct 30 15:57:17.400402 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.400392 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aafde56-f3e7-415b-83b7-98c52e390add" containerName="revoke-keys" Oct 30 15:57:17.404475 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.404448 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.426574 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.426537 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ncks\" (UniqueName: \"kubernetes.io/projected/06a40a3f-28dd-4ff7-ae48-33c72d796a21-kube-api-access-6ncks\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.426783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.426691 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.427137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.427107 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.432463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.432433 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq"] Oct 30 15:57:17.528655 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.528611 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.529231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.528672 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ncks\" (UniqueName: \"kubernetes.io/projected/06a40a3f-28dd-4ff7-ae48-33c72d796a21-kube-api-access-6ncks\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.529231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.528796 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.529423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.529400 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.531265 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.531240 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.555246 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.555213 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ncks\" (UniqueName: \"kubernetes.io/projected/06a40a3f-28dd-4ff7-ae48-33c72d796a21-kube-api-access-6ncks\") pod \"maas-api-revoke-sub-fa2ac439-gxfwq\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.724878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.722136 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:17.803889 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.803810 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" Oct 30 15:57:17.804150 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.803838 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl" event={"ID":"9aafde56-f3e7-415b-83b7-98c52e390add","Type":"ContainerDied","Data":"8367910d08572894ae835abe9cceec86713672daa7fd4643b871dbe88b57a0ff"} Oct 30 15:57:17.804311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.804295 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8367910d08572894ae835abe9cceec86713672daa7fd4643b871dbe88b57a0ff" Oct 30 15:57:17.810841 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.806238 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" event={"ID":"faca6252-0610-4f50-aed9-bd07ef8c7938","Type":"ContainerStarted","Data":"03b7e97207d6beedfca8b6dbbe4dec5bdade941bec9679a39af4712a30ef89ee"} Oct 30 15:57:17.810841 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.806296 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" event={"ID":"faca6252-0610-4f50-aed9-bd07ef8c7938","Type":"ContainerStarted","Data":"32ea1becb12d1a177b304a92abd9e122fb5800e8f41042bce5506261f9c2f601"} Oct 30 15:57:17.866009 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:17.865943 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" podStartSLOduration=1.8659180389999999 podStartE2EDuration="1.865918039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:17.858656755 +0000 UTC m=+1146.557929912" watchObservedRunningTime="2026-10-30 15:57:17.865918039 +0000 UTC m=+1146.565191196" Oct 30 15:57:18.117221 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:18.117184 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a40a3f_28dd_4ff7_ae48_33c72d796a21.slice/crio-b96b141c5323bb00854b54ede70f94fbccc682daa49ceeda22159d5fae96aa66 WatchSource:0}: Error finding container b96b141c5323bb00854b54ede70f94fbccc682daa49ceeda22159d5fae96aa66: Status 404 returned error can't find the container with id b96b141c5323bb00854b54ede70f94fbccc682daa49ceeda22159d5fae96aa66 Oct 30 15:57:18.120046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:18.119814 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq"] Oct 30 15:57:18.816749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:18.816664 4401 generic.go:356] "Generic (PLEG): container finished" podID="faca6252-0610-4f50-aed9-bd07ef8c7938" containerID="03b7e97207d6beedfca8b6dbbe4dec5bdade941bec9679a39af4712a30ef89ee" exitCode=0 Oct 30 15:57:18.816749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:18.816737 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" event={"ID":"faca6252-0610-4f50-aed9-bd07ef8c7938","Type":"ContainerDied","Data":"03b7e97207d6beedfca8b6dbbe4dec5bdade941bec9679a39af4712a30ef89ee"} Oct 30 15:57:18.818532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:18.818497 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" event={"ID":"06a40a3f-28dd-4ff7-ae48-33c72d796a21","Type":"ContainerStarted","Data":"17c47db85daf615598b61c550b9a26e0f628226c4c996769d9bfad6816579d47"} Oct 30 15:57:18.818683 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:18.818557 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" event={"ID":"06a40a3f-28dd-4ff7-ae48-33c72d796a21","Type":"ContainerStarted","Data":"b96b141c5323bb00854b54ede70f94fbccc682daa49ceeda22159d5fae96aa66"} Oct 30 15:57:18.907879 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:18.907793 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" podStartSLOduration=1.907768704 podStartE2EDuration="1.907768704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:18.904179272 +0000 UTC m=+1147.603452423" watchObservedRunningTime="2026-10-30 15:57:18.907768704 +0000 UTC m=+1147.607041863" Oct 30 15:57:19.829608 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:19.829566 4401 generic.go:356] "Generic (PLEG): container finished" podID="06a40a3f-28dd-4ff7-ae48-33c72d796a21" containerID="17c47db85daf615598b61c550b9a26e0f628226c4c996769d9bfad6816579d47" exitCode=0 Oct 30 15:57:19.830129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:19.829659 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" event={"ID":"06a40a3f-28dd-4ff7-ae48-33c72d796a21","Type":"ContainerDied","Data":"17c47db85daf615598b61c550b9a26e0f628226c4c996769d9bfad6816579d47"} Oct 30 15:57:20.025728 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.025687 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:20.074203 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.074160 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faca6252-0610-4f50-aed9-bd07ef8c7938-kube-api-access-xpbl8\" (UniqueName: \"kubernetes.io/projected/faca6252-0610-4f50-aed9-bd07ef8c7938-kube-api-access-xpbl8\") pod \"faca6252-0610-4f50-aed9-bd07ef8c7938\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " Oct 30 15:57:20.074355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.074264 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-xks-ca\") pod \"faca6252-0610-4f50-aed9-bd07ef8c7938\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " Oct 30 15:57:20.074425 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.074387 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-openshift-ca\") pod \"faca6252-0610-4f50-aed9-bd07ef8c7938\" (UID: \"faca6252-0610-4f50-aed9-bd07ef8c7938\") " Oct 30 15:57:20.074703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.074672 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-openshift-ca" pod "faca6252-0610-4f50-aed9-bd07ef8c7938" (UID: "faca6252-0610-4f50-aed9-bd07ef8c7938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:20.075035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.075010 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:20.076693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.076656 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faca6252-0610-4f50-aed9-bd07ef8c7938-kube-api-access-xpbl8" pod "faca6252-0610-4f50-aed9-bd07ef8c7938" (UID: "faca6252-0610-4f50-aed9-bd07ef8c7938"). InnerVolumeSpecName "kube-api-access-xpbl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:20.076969 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.076944 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-xks-ca" pod "faca6252-0610-4f50-aed9-bd07ef8c7938" (UID: "faca6252-0610-4f50-aed9-bd07ef8c7938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:20.176797 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.176758 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbl8\" (UniqueName: \"kubernetes.io/projected/faca6252-0610-4f50-aed9-bd07ef8c7938-kube-api-access-xpbl8\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:20.176797 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.176792 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faca6252-0610-4f50-aed9-bd07ef8c7938-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:20.841579 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.841549 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" Oct 30 15:57:20.842034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.841575 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm" event={"ID":"faca6252-0610-4f50-aed9-bd07ef8c7938","Type":"ContainerDied","Data":"32ea1becb12d1a177b304a92abd9e122fb5800e8f41042bce5506261f9c2f601"} Oct 30 15:57:20.842034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:20.841622 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32ea1becb12d1a177b304a92abd9e122fb5800e8f41042bce5506261f9c2f601" Oct 30 15:57:21.018313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.018285 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:21.088074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.088024 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-openshift-ca\") pod \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " Oct 30 15:57:21.088074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.088088 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-xks-ca\") pod \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " Oct 30 15:57:21.088292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.088216 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a40a3f-28dd-4ff7-ae48-33c72d796a21-kube-api-access-6ncks\" (UniqueName: \"kubernetes.io/projected/06a40a3f-28dd-4ff7-ae48-33c72d796a21-kube-api-access-6ncks\") pod \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\" (UID: \"06a40a3f-28dd-4ff7-ae48-33c72d796a21\") " Oct 30 15:57:21.088470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.088440 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-openshift-ca" pod "06a40a3f-28dd-4ff7-ae48-33c72d796a21" (UID: "06a40a3f-28dd-4ff7-ae48-33c72d796a21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:21.088886 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.088859 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:21.090527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.090499 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-xks-ca" pod "06a40a3f-28dd-4ff7-ae48-33c72d796a21" (UID: "06a40a3f-28dd-4ff7-ae48-33c72d796a21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:21.090645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.090544 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a40a3f-28dd-4ff7-ae48-33c72d796a21-kube-api-access-6ncks" pod "06a40a3f-28dd-4ff7-ae48-33c72d796a21" (UID: "06a40a3f-28dd-4ff7-ae48-33c72d796a21"). InnerVolumeSpecName "kube-api-access-6ncks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:21.189736 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.189695 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ncks\" (UniqueName: \"kubernetes.io/projected/06a40a3f-28dd-4ff7-ae48-33c72d796a21-kube-api-access-6ncks\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:21.189736 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.189731 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06a40a3f-28dd-4ff7-ae48-33c72d796a21-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:21.852503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.852471 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" Oct 30 15:57:21.852926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.852516 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq" event={"ID":"06a40a3f-28dd-4ff7-ae48-33c72d796a21","Type":"ContainerDied","Data":"b96b141c5323bb00854b54ede70f94fbccc682daa49ceeda22159d5fae96aa66"} Oct 30 15:57:21.852926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:21.852568 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b96b141c5323bb00854b54ede70f94fbccc682daa49ceeda22159d5fae96aa66" Oct 30 15:57:21.969433 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:57:21.969397 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:57:24.317998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:24.317938 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:24.533031 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:24.532972 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:24.555997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:24.555923 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:24.691985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:24.691926 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:24.748985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:24.748925 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.044920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.039743 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.160851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.160777 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.355921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.353392 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.468164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.358538 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.468164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.359124 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:57:25.539226 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.533650 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.655907 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.571064 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:57:25.755901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.709929 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.757290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.756717 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:25.767697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.761441 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:57:25.767697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.761521 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:57:25.767697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.761558 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:57:25.767697 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:25.761597 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:26.144917 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.075794 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:26.238781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.165195 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:57:26.238781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.166459 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:26.238781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.167996 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:57:26.406913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.378983 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:57:26.450121 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.424267 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:26.450121 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.441857 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:57:26.526851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.501921 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:26.569634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.569060 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:26.651851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.651536 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:26.911853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.876597 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:26.979387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:26.915997 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:57:27.223020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.201918 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:57:27.223020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.202373 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:57:27.223020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.210147 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": context deadline exceeded" Oct 30 15:57:27.372134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.371997 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:27.623972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.566766 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:27.689906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.668182 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:27.689906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.669230 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:27.933922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:27.902795 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded" Oct 30 15:57:28.131875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.125580 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 15:57:28.274352 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.274212 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:57:28.295883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.278947 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:57:28.320870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.314057 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:28.664850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.663585 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:57:28.702064 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.702011 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:28.713863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.702896 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:28.713863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.703985 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-b7d29" podUID="dd5a7553-8377-498b-ab94-0fb53d0c83fb" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.25:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:28.962905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.936445 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:28.968856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:28.968553 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:29.134040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.133987 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc"] Oct 30 15:57:29.158404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.158354 4401 cpu_manager.go:414] "containerMap: removing container" podUID="faca6252-0610-4f50-aed9-bd07ef8c7938" containerName="revoke-keys" Oct 30 15:57:29.161931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.161882 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faca6252-0610-4f50-aed9-bd07ef8c7938" containerName="revoke-keys" Oct 30 15:57:29.165913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.165882 4401 cpu_manager.go:414] "containerMap: removing container" podUID="06a40a3f-28dd-4ff7-ae48-33c72d796a21" containerName="revoke-keys" Oct 30 15:57:29.166179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.166161 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a40a3f-28dd-4ff7-ae48-33c72d796a21" containerName="revoke-keys" Oct 30 15:57:29.169866 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.165362 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:29.206853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.186120 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="faca6252-0610-4f50-aed9-bd07ef8c7938" containerName="revoke-keys" Oct 30 15:57:29.207245 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.207218 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a40a3f-28dd-4ff7-ae48-33c72d796a21" containerName="revoke-keys" Oct 30 15:57:29.366005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.365946 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 15:57:29.366369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.366334 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:29.380030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.379979 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:29.393337 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.379993 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:29.408079 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.408034 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.493611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.493577 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:57:29.494048 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.494030 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:57:29.570382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.553107 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:57:29.570382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.568119 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.570382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.568189 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c99pv\" (UniqueName: \"kubernetes.io/projected/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-kube-api-access-c99pv\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.570382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.568505 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.582536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.582504 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:57:29.667047 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.666935 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:29.669938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.669899 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.700549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.700513 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.702003 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.701975 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c99pv\" (UniqueName: \"kubernetes.io/projected/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-kube-api-access-c99pv\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.720853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.701598 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.723926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.700258 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc"] Oct 30 15:57:29.724170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.711083 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:29.724656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.711119 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:57:29.725305 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.711476 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.759171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.759131 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb"] Oct 30 15:57:29.765809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.765766 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:29.786250 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.786214 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c99pv\" (UniqueName: \"kubernetes.io/projected/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-kube-api-access-c99pv\") pod \"maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.896464 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.896424 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb"] Oct 30 15:57:29.911851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.907926 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:29.915850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.912173 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxlh\" (UniqueName: \"kubernetes.io/projected/89b35df3-109f-446b-8115-c12d691afd6d-kube-api-access-4nxlh\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:29.916190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.916164 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:29.954473 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.954126 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:29.954754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:29.954716 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:30.022418 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.022352 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:30.023514 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.023472 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxlh\" (UniqueName: \"kubernetes.io/projected/89b35df3-109f-446b-8115-c12d691afd6d-kube-api-access-4nxlh\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:30.060266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.060113 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:30.060266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.060241 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:30.085995 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.080370 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxlh\" (UniqueName: \"kubernetes.io/projected/89b35df3-109f-446b-8115-c12d691afd6d-kube-api-access-4nxlh\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:30.287462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.287331 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6a50dd6-nmwvb\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:30.383883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.380217 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:30.397847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.393611 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:30.397847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.396382 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:30.422058 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.418244 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:30.526889 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.514417 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:57:30.526889 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.516537 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:30.544013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.543650 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:30.676856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.674359 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": context deadline exceeded" Oct 30 15:57:30.754012 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.732693 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:30.788893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.765956 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 15:57:30.964146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.964110 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:57:30.965515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.965483 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:57:30.971023 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.970996 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:57:30.995983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.995948 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:57:30.996175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:30.996017 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:57:31.001730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.001699 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:57:31.091943 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.091914 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc"] Oct 30 15:57:31.372175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.372079 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb"] Oct 30 15:57:31.448689 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:31.448648 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b35df3_109f_446b_8115_c12d691afd6d.slice/crio-adea43e674b0c7b94ffc8639d7f8d07b1ea9fecf3f95be39c645f1ccacae561d WatchSource:0}: Error finding container adea43e674b0c7b94ffc8639d7f8d07b1ea9fecf3f95be39c645f1ccacae561d: Status 404 returned error can't find the container with id adea43e674b0c7b94ffc8639d7f8d07b1ea9fecf3f95be39c645f1ccacae561d Oct 30 15:57:31.573852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.570613 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx"] Oct 30 15:57:31.577847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.577447 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.638126 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.638031 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.638330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.638155 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wvd\" (UniqueName: \"kubernetes.io/projected/f78194cb-cbae-4084-bf47-8d496b6028ff-kube-api-access-z6wvd\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.638330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.638281 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.638852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.638806 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx"] Oct 30 15:57:31.739932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.739895 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wvd\" (UniqueName: \"kubernetes.io/projected/f78194cb-cbae-4084-bf47-8d496b6028ff-kube-api-access-z6wvd\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.740361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.739995 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.740361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.740105 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.740695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.740664 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.742616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.742578 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.766131 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.766068 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wvd\" (UniqueName: \"kubernetes.io/projected/f78194cb-cbae-4084-bf47-8d496b6028ff-kube-api-access-z6wvd\") pod \"maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.957928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.957810 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:31.991761 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.991720 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" event={"ID":"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5","Type":"ContainerStarted","Data":"ba25de8e19066d90e73b11558bee8a7d2fe912e9f64b2ee3550a0a4f368c2f11"} Oct 30 15:57:31.991942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.991789 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" event={"ID":"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5","Type":"ContainerStarted","Data":"95a8313c4f95726a98bbf28065a826209452b096b8476c7f53034b76e53584a4"} Oct 30 15:57:31.997026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.994275 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" event={"ID":"89b35df3-109f-446b-8115-c12d691afd6d","Type":"ContainerStarted","Data":"bf4e08c283b7905f33c8a9661939eabc9eac5e7e1634950af98b1d3ad433ba11"} Oct 30 15:57:31.997026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:31.994388 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" event={"ID":"89b35df3-109f-446b-8115-c12d691afd6d","Type":"ContainerStarted","Data":"adea43e674b0c7b94ffc8639d7f8d07b1ea9fecf3f95be39c645f1ccacae561d"} Oct 30 15:57:32.045276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:32.044196 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" podStartSLOduration=4.044177726 podStartE2EDuration="4.044177726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:32.037240665 +0000 UTC m=+1160.736513816" watchObservedRunningTime="2026-10-30 15:57:32.044177726 +0000 UTC m=+1160.743450883" Oct 30 15:57:32.305951 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:32.305889 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" podStartSLOduration=4.305872069 podStartE2EDuration="4.305872069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:32.092116378 +0000 UTC m=+1160.791389535" watchObservedRunningTime="2026-10-30 15:57:32.305872069 +0000 UTC m=+1161.005145227" Oct 30 15:57:32.308317 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:32.308284 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78194cb_cbae_4084_bf47_8d496b6028ff.slice/crio-a35d0fc643b09f1bf499d7ed534d5e7eb554becd7d4f5f671235e088ad7d424e WatchSource:0}: Error finding container a35d0fc643b09f1bf499d7ed534d5e7eb554becd7d4f5f671235e088ad7d424e: Status 404 returned error can't find the container with id a35d0fc643b09f1bf499d7ed534d5e7eb554becd7d4f5f671235e088ad7d424e Oct 30 15:57:32.311099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:32.311075 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx"] Oct 30 15:57:33.005678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.005569 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" event={"ID":"f78194cb-cbae-4084-bf47-8d496b6028ff","Type":"ContainerStarted","Data":"8bfe222badf6fc046d2c0aa72f027ad3a4160ef8eef4633f28603eb0c582945c"} Oct 30 15:57:33.005678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.005672 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" event={"ID":"f78194cb-cbae-4084-bf47-8d496b6028ff","Type":"ContainerStarted","Data":"a35d0fc643b09f1bf499d7ed534d5e7eb554becd7d4f5f671235e088ad7d424e"} Oct 30 15:57:33.007440 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.007413 4401 generic.go:356] "Generic (PLEG): container finished" podID="c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" containerID="ba25de8e19066d90e73b11558bee8a7d2fe912e9f64b2ee3550a0a4f368c2f11" exitCode=0 Oct 30 15:57:33.007540 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.007462 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" event={"ID":"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5","Type":"ContainerDied","Data":"ba25de8e19066d90e73b11558bee8a7d2fe912e9f64b2ee3550a0a4f368c2f11"} Oct 30 15:57:33.009289 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.009261 4401 generic.go:356] "Generic (PLEG): container finished" podID="89b35df3-109f-446b-8115-c12d691afd6d" containerID="bf4e08c283b7905f33c8a9661939eabc9eac5e7e1634950af98b1d3ad433ba11" exitCode=0 Oct 30 15:57:33.009388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.009378 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" event={"ID":"89b35df3-109f-446b-8115-c12d691afd6d","Type":"ContainerDied","Data":"bf4e08c283b7905f33c8a9661939eabc9eac5e7e1634950af98b1d3ad433ba11"} Oct 30 15:57:33.053378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.053312 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" podStartSLOduration=2.053297489 podStartE2EDuration="2.053297489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:33.050102501 +0000 UTC m=+1161.749375657" watchObservedRunningTime="2026-10-30 15:57:33.053297489 +0000 UTC m=+1161.752570646" Oct 30 15:57:33.285839 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:33.285736 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj"] Oct 30 15:57:34.021266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.021224 4401 generic.go:356] "Generic (PLEG): container finished" podID="f78194cb-cbae-4084-bf47-8d496b6028ff" containerID="8bfe222badf6fc046d2c0aa72f027ad3a4160ef8eef4633f28603eb0c582945c" exitCode=0 Oct 30 15:57:34.021742 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.021309 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" event={"ID":"f78194cb-cbae-4084-bf47-8d496b6028ff","Type":"ContainerDied","Data":"8bfe222badf6fc046d2c0aa72f027ad3a4160ef8eef4633f28603eb0c582945c"} Oct 30 15:57:34.224326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.224293 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:34.255634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.255606 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:34.292400 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.292303 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-openshift-ca\") pod \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " Oct 30 15:57:34.292584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.292462 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-xks-ca\") pod \"89b35df3-109f-446b-8115-c12d691afd6d\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " Oct 30 15:57:34.292584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.292512 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b35df3-109f-446b-8115-c12d691afd6d-kube-api-access-4nxlh\" (UniqueName: \"kubernetes.io/projected/89b35df3-109f-446b-8115-c12d691afd6d-kube-api-access-4nxlh\") pod \"89b35df3-109f-446b-8115-c12d691afd6d\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " Oct 30 15:57:34.292745 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.292666 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-kube-api-access-c99pv\" (UniqueName: \"kubernetes.io/projected/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-kube-api-access-c99pv\") pod \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " Oct 30 15:57:34.292745 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.292718 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-xks-ca\") pod \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\" (UID: \"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5\") " Oct 30 15:57:34.292855 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.292729 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-openshift-ca" pod "c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" (UID: "c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:34.292855 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.292835 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-openshift-ca\") pod \"89b35df3-109f-446b-8115-c12d691afd6d\" (UID: \"89b35df3-109f-446b-8115-c12d691afd6d\") " Oct 30 15:57:34.293483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.293435 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-openshift-ca" pod "89b35df3-109f-446b-8115-c12d691afd6d" (UID: "89b35df3-109f-446b-8115-c12d691afd6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:34.293602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.293525 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:34.295543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.295511 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-xks-ca" pod "c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" (UID: "c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:34.296491 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.296458 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-xks-ca" pod "89b35df3-109f-446b-8115-c12d691afd6d" (UID: "89b35df3-109f-446b-8115-c12d691afd6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:34.296491 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.296457 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b35df3-109f-446b-8115-c12d691afd6d-kube-api-access-4nxlh" pod "89b35df3-109f-446b-8115-c12d691afd6d" (UID: "89b35df3-109f-446b-8115-c12d691afd6d"). InnerVolumeSpecName "kube-api-access-4nxlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:34.297987 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.297956 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-kube-api-access-c99pv" pod "c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" (UID: "c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5"). InnerVolumeSpecName "kube-api-access-c99pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:34.394782 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.394740 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c99pv\" (UniqueName: \"kubernetes.io/projected/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-kube-api-access-c99pv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:34.394782 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.394776 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:34.394782 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.394786 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:34.395074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.394795 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b35df3-109f-446b-8115-c12d691afd6d-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:34.395074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:34.394804 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nxlh\" (UniqueName: \"kubernetes.io/projected/89b35df3-109f-446b-8115-c12d691afd6d-kube-api-access-4nxlh\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:35.033567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.033536 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" Oct 30 15:57:35.034057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.033571 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc" event={"ID":"c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5","Type":"ContainerDied","Data":"95a8313c4f95726a98bbf28065a826209452b096b8476c7f53034b76e53584a4"} Oct 30 15:57:35.034057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.033620 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a8313c4f95726a98bbf28065a826209452b096b8476c7f53034b76e53584a4" Oct 30 15:57:35.035255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.035232 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" Oct 30 15:57:35.035361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.035275 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb" event={"ID":"89b35df3-109f-446b-8115-c12d691afd6d","Type":"ContainerDied","Data":"adea43e674b0c7b94ffc8639d7f8d07b1ea9fecf3f95be39c645f1ccacae561d"} Oct 30 15:57:35.035361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.035317 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adea43e674b0c7b94ffc8639d7f8d07b1ea9fecf3f95be39c645f1ccacae561d" Oct 30 15:57:35.220073 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.220035 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:35.454699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.454660 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-xks-ca\") pod \"f78194cb-cbae-4084-bf47-8d496b6028ff\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " Oct 30 15:57:35.454960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.454937 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f78194cb-cbae-4084-bf47-8d496b6028ff-kube-api-access-z6wvd\" (UniqueName: \"kubernetes.io/projected/f78194cb-cbae-4084-bf47-8d496b6028ff-kube-api-access-z6wvd\") pod \"f78194cb-cbae-4084-bf47-8d496b6028ff\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " Oct 30 15:57:35.455051 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.455035 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-openshift-ca\") pod \"f78194cb-cbae-4084-bf47-8d496b6028ff\" (UID: \"f78194cb-cbae-4084-bf47-8d496b6028ff\") " Oct 30 15:57:35.455495 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.455459 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-openshift-ca" pod "f78194cb-cbae-4084-bf47-8d496b6028ff" (UID: "f78194cb-cbae-4084-bf47-8d496b6028ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:35.455887 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.455816 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:35.457252 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.457227 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-xks-ca" pod "f78194cb-cbae-4084-bf47-8d496b6028ff" (UID: "f78194cb-cbae-4084-bf47-8d496b6028ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:35.457394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.457370 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f78194cb-cbae-4084-bf47-8d496b6028ff-kube-api-access-z6wvd" pod "f78194cb-cbae-4084-bf47-8d496b6028ff" (UID: "f78194cb-cbae-4084-bf47-8d496b6028ff"). InnerVolumeSpecName "kube-api-access-z6wvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:35.556795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.556758 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f78194cb-cbae-4084-bf47-8d496b6028ff-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:35.556795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:35.556791 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6wvd\" (UniqueName: \"kubernetes.io/projected/f78194cb-cbae-4084-bf47-8d496b6028ff-kube-api-access-z6wvd\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:36.045627 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:36.045594 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" Oct 30 15:57:36.045627 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:36.045597 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx" event={"ID":"f78194cb-cbae-4084-bf47-8d496b6028ff","Type":"ContainerDied","Data":"a35d0fc643b09f1bf499d7ed534d5e7eb554becd7d4f5f671235e088ad7d424e"} Oct 30 15:57:36.046121 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:36.045664 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35d0fc643b09f1bf499d7ed534d5e7eb554becd7d4f5f671235e088ad7d424e" Oct 30 15:57:36.288416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:36.288377 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj"] Oct 30 15:57:36.295378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:36.295344 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79"] Oct 30 15:57:39.027947 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:39.027904 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78"] Oct 30 15:57:39.053098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:39.053066 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb"] Oct 30 15:57:40.249636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.249592 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s"] Oct 30 15:57:40.250473 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250452 4401 cpu_manager.go:414] "containerMap: removing container" podUID="89b35df3-109f-446b-8115-c12d691afd6d" containerName="revoke-keys" Oct 30 15:57:40.250473 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250474 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b35df3-109f-446b-8115-c12d691afd6d" containerName="revoke-keys" Oct 30 15:57:40.250584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250490 4401 cpu_manager.go:414] "containerMap: removing container" podUID="f78194cb-cbae-4084-bf47-8d496b6028ff" containerName="revoke-keys" Oct 30 15:57:40.250584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250495 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f78194cb-cbae-4084-bf47-8d496b6028ff" containerName="revoke-keys" Oct 30 15:57:40.250584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250530 4401 cpu_manager.go:414] "containerMap: removing container" podUID="c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" containerName="revoke-keys" Oct 30 15:57:40.250584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250536 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" containerName="revoke-keys" Oct 30 15:57:40.250709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250642 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" containerName="revoke-keys" Oct 30 15:57:40.250709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250656 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b35df3-109f-446b-8115-c12d691afd6d" containerName="revoke-keys" Oct 30 15:57:40.250709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.250664 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="f78194cb-cbae-4084-bf47-8d496b6028ff" containerName="revoke-keys" Oct 30 15:57:40.254516 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.254491 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.269293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.269262 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr"] Oct 30 15:57:40.274019 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.273985 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.278890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.278853 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s"] Oct 30 15:57:40.280273 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.280239 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:57:40.280714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.280339 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:57:40.280714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.280386 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:57:40.280714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.280392 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:57:40.305164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.305127 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr"] Oct 30 15:57:40.351151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.351113 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.351314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.351218 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.351314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.351290 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.351430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.351323 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vcl\" (UniqueName: \"kubernetes.io/projected/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-kube-api-access-f5vcl\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.351543 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.351519 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.351590 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.351575 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb2jl\" (UniqueName: \"kubernetes.io/projected/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-kube-api-access-qb2jl\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.452622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.452582 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.452783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.452634 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vcl\" (UniqueName: \"kubernetes.io/projected/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-kube-api-access-f5vcl\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.452783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.452715 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.452783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.452759 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb2jl\" (UniqueName: \"kubernetes.io/projected/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-kube-api-access-qb2jl\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.452996 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.452874 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.452996 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.452933 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.453513 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.453489 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.453633 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.453605 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.455095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.455076 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.455226 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.455207 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.478231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.478193 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vcl\" (UniqueName: \"kubernetes.io/projected/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-kube-api-access-f5vcl\") pod \"maas-api-revoke-sub-5d818911-ct67s\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.478453 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.478432 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb2jl\" (UniqueName: \"kubernetes.io/projected/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-kube-api-access-qb2jl\") pod \"maas-api-revoke-sub-9a0931d3-zkgmr\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.572352 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.572269 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:40.598672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.598643 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:40.949406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.949310 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s"] Oct 30 15:57:40.985046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:40.977217 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr"] Oct 30 15:57:40.985046 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:40.984195 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4dfe6b_39ff_4a6e_a6c9_32e24a8edd45.slice/crio-61f92e3cb73dce5216bf46470dcf3e2f37fd146e1f744ea2f866ffccb8bae9af WatchSource:0}: Error finding container 61f92e3cb73dce5216bf46470dcf3e2f37fd146e1f744ea2f866ffccb8bae9af: Status 404 returned error can't find the container with id 61f92e3cb73dce5216bf46470dcf3e2f37fd146e1f744ea2f866ffccb8bae9af Oct 30 15:57:41.050561 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.050523 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn"] Oct 30 15:57:41.056571 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.056546 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.073711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.073682 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn"] Oct 30 15:57:41.148076 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.148038 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" event={"ID":"26b94c73-08da-4cb8-bd5f-1e5b6eef176f","Type":"ContainerStarted","Data":"7a4970e5e5d1b0594d63fac88b0267bbca4d363b23470acce4f75c6fd7bffa3b"} Oct 30 15:57:41.149933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.149893 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" event={"ID":"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45","Type":"ContainerStarted","Data":"61f92e3cb73dce5216bf46470dcf3e2f37fd146e1f744ea2f866ffccb8bae9af"} Oct 30 15:57:41.174128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.174092 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.174311 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.174283 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf72h\" (UniqueName: \"kubernetes.io/projected/865178fe-5865-488d-9416-452e1b9e2aef-kube-api-access-vf72h\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.174379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.174346 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.275571 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.275534 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf72h\" (UniqueName: \"kubernetes.io/projected/865178fe-5865-488d-9416-452e1b9e2aef-kube-api-access-vf72h\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.275999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.275591 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.275999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.275742 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.276474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.276448 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.278015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.277994 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.301597 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.301562 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf72h\" (UniqueName: \"kubernetes.io/projected/865178fe-5865-488d-9416-452e1b9e2aef-kube-api-access-vf72h\") pod \"maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.382229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.382186 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:41.698899 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:41.698812 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865178fe_5865_488d_9416_452e1b9e2aef.slice/crio-0930b979ba11eefd169ec456bf11497dbc8d1714b9d1a2323b4057549f018ec9 WatchSource:0}: Error finding container 0930b979ba11eefd169ec456bf11497dbc8d1714b9d1a2323b4057549f018ec9: Status 404 returned error can't find the container with id 0930b979ba11eefd169ec456bf11497dbc8d1714b9d1a2323b4057549f018ec9 Oct 30 15:57:41.701351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.701324 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn"] Oct 30 15:57:41.737409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:41.737368 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw"] Oct 30 15:57:42.161754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.161715 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" event={"ID":"865178fe-5865-488d-9416-452e1b9e2aef","Type":"ContainerStarted","Data":"0930b979ba11eefd169ec456bf11497dbc8d1714b9d1a2323b4057549f018ec9"} Oct 30 15:57:42.163836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.163797 4401 generic.go:356] "Generic (PLEG): container finished" podID="4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" containerID="60eb9f5049d0a0fcd63a6e63e6d2e8d79974fb139cc0b5d95179b85e3c5a545e" exitCode=0 Oct 30 15:57:42.163915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.163865 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" event={"ID":"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45","Type":"ContainerDied","Data":"60eb9f5049d0a0fcd63a6e63e6d2e8d79974fb139cc0b5d95179b85e3c5a545e"} Oct 30 15:57:42.165901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.165867 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" event={"ID":"26b94c73-08da-4cb8-bd5f-1e5b6eef176f","Type":"ContainerStarted","Data":"66a4889cafb2606610dc69bae2db186f00e7cd6a888e3fb3d440625822595b81"} Oct 30 15:57:42.722133 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.722095 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7"] Oct 30 15:57:42.727253 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.727220 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.751295 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.751257 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7"] Oct 30 15:57:42.824178 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824136 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30a03f0-6167-47ce-8897-47c6de0bfae8-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824265 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824314 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rqj\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-kube-api-access-68rqj\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824351 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824497 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824597 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824664 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824741 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824728 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.824790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.824778 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952219 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952349 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rqj\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-kube-api-access-68rqj\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952425 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952512 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952632 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952703 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952853 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.952948 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.953863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.953084 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30a03f0-6167-47ce-8897-47c6de0bfae8-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.954498 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.954256 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30a03f0-6167-47ce-8897-47c6de0bfae8-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.955199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.954665 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.971775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.971700 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.972268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.972192 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.972530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.972511 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.993245 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.988598 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:42.999578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:42.999536 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:43.009387 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.000974 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rqj\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-kube-api-access-68rqj\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:43.017471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.015814 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:43.051879 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.051844 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:43.183435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.183310 4401 generic.go:356] "Generic (PLEG): container finished" podID="865178fe-5865-488d-9416-452e1b9e2aef" containerID="588a59429711851209df5731b74d2b2c3fcca9b726cdf9bc7f39cbfeaaf27540" exitCode=0 Oct 30 15:57:43.183435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.183370 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" event={"ID":"865178fe-5865-488d-9416-452e1b9e2aef","Type":"ContainerDied","Data":"588a59429711851209df5731b74d2b2c3fcca9b726cdf9bc7f39cbfeaaf27540"} Oct 30 15:57:43.185622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.185594 4401 generic.go:356] "Generic (PLEG): container finished" podID="26b94c73-08da-4cb8-bd5f-1e5b6eef176f" containerID="66a4889cafb2606610dc69bae2db186f00e7cd6a888e3fb3d440625822595b81" exitCode=0 Oct 30 15:57:43.185749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.185681 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" event={"ID":"26b94c73-08da-4cb8-bd5f-1e5b6eef176f","Type":"ContainerDied","Data":"66a4889cafb2606610dc69bae2db186f00e7cd6a888e3fb3d440625822595b81"} Oct 30 15:57:43.350106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.350067 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:43.403781 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:43.403743 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30a03f0_6167_47ce_8897_47c6de0bfae8.slice/crio-93d370ddb0f403b6571ee4c451071a0dec26d207376de4ce7ca8c0e989452f27 WatchSource:0}: Error finding container 93d370ddb0f403b6571ee4c451071a0dec26d207376de4ce7ca8c0e989452f27: Status 404 returned error can't find the container with id 93d370ddb0f403b6571ee4c451071a0dec26d207376de4ce7ca8c0e989452f27 Oct 30 15:57:43.406722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.406692 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:43.407460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.407420 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:57:43.407660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.407638 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:57:43.408304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.408272 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:57:43.408880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.408809 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7"] Oct 30 15:57:43.462138 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.462102 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-openshift-ca\") pod \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " Oct 30 15:57:43.462592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.462566 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-xks-ca\") pod \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " Oct 30 15:57:43.462968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.462805 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-kube-api-access-f5vcl\" (UniqueName: \"kubernetes.io/projected/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-kube-api-access-f5vcl\") pod \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\" (UID: \"26b94c73-08da-4cb8-bd5f-1e5b6eef176f\") " Oct 30 15:57:43.463958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.463930 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-openshift-ca" pod "26b94c73-08da-4cb8-bd5f-1e5b6eef176f" (UID: "26b94c73-08da-4cb8-bd5f-1e5b6eef176f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:43.465990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.465960 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-xks-ca" pod "26b94c73-08da-4cb8-bd5f-1e5b6eef176f" (UID: "26b94c73-08da-4cb8-bd5f-1e5b6eef176f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:43.469400 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.469357 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-kube-api-access-f5vcl" pod "26b94c73-08da-4cb8-bd5f-1e5b6eef176f" (UID: "26b94c73-08da-4cb8-bd5f-1e5b6eef176f"). InnerVolumeSpecName "kube-api-access-f5vcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:43.565008 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.564912 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-kube-api-access-qb2jl\" (UniqueName: \"kubernetes.io/projected/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-kube-api-access-qb2jl\") pod \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " Oct 30 15:57:43.565195 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.565049 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-openshift-ca\") pod \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " Oct 30 15:57:43.565195 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.565114 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-xks-ca\") pod \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\" (UID: \"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45\") " Oct 30 15:57:43.565415 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.565387 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-openshift-ca" pod "4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" (UID: "4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:43.565878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.565812 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:43.565964 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.565880 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:43.565964 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.565891 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:43.565964 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.565900 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5vcl\" (UniqueName: \"kubernetes.io/projected/26b94c73-08da-4cb8-bd5f-1e5b6eef176f-kube-api-access-f5vcl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:43.567135 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.567106 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-kube-api-access-qb2jl" pod "4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" (UID: "4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45"). InnerVolumeSpecName "kube-api-access-qb2jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:43.567255 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.567158 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-xks-ca" pod "4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" (UID: "4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:43.668013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.667973 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:43.668013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:43.668011 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb2jl\" (UniqueName: \"kubernetes.io/projected/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45-kube-api-access-qb2jl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:44.029245 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.029184 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" podUID="c28756ca-98c8-47ec-a504-689b12e1052e" containerName="payload-processing" containerID="cri-o://d1611b097b1bdaea618a5b84be286cbd6b6e1cb117341611b78f3afc02b15311" gracePeriod=25 Oct 30 15:57:44.054002 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.053902 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" podUID="9ab32413-1c61-458b-87f9-4696ae69aba9" containerName="payload-pre-processing" containerID="cri-o://98d12d97b494528caf50602dd633980a7544418710a24241a42e618bfd54ab1c" gracePeriod=25 Oct 30 15:57:44.076748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.076704 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv"] Oct 30 15:57:44.082852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.079441 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d41d55d-72dtv"] Oct 30 15:57:44.196680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.196586 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" Oct 30 15:57:44.196680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.196603 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s" event={"ID":"26b94c73-08da-4cb8-bd5f-1e5b6eef176f","Type":"ContainerDied","Data":"7a4970e5e5d1b0594d63fac88b0267bbca4d363b23470acce4f75c6fd7bffa3b"} Oct 30 15:57:44.196940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.196696 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a4970e5e5d1b0594d63fac88b0267bbca4d363b23470acce4f75c6fd7bffa3b" Oct 30 15:57:44.198695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.198657 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" event={"ID":"a30a03f0-6167-47ce-8897-47c6de0bfae8","Type":"ContainerStarted","Data":"2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51"} Oct 30 15:57:44.198807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.198719 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" event={"ID":"a30a03f0-6167-47ce-8897-47c6de0bfae8","Type":"ContainerStarted","Data":"93d370ddb0f403b6571ee4c451071a0dec26d207376de4ce7ca8c0e989452f27"} Oct 30 15:57:44.200569 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.200539 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" event={"ID":"4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45","Type":"ContainerDied","Data":"61f92e3cb73dce5216bf46470dcf3e2f37fd146e1f744ea2f866ffccb8bae9af"} Oct 30 15:57:44.200661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.200587 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61f92e3cb73dce5216bf46470dcf3e2f37fd146e1f744ea2f866ffccb8bae9af" Oct 30 15:57:44.200696 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.200682 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr" Oct 30 15:57:44.262860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.255685 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podStartSLOduration=2.255662482 podStartE2EDuration="2.255662482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:44.254315848 +0000 UTC m=+1172.953589006" watchObservedRunningTime="2026-10-30 15:57:44.255662482 +0000 UTC m=+1172.954935640" Oct 30 15:57:44.420800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.420768 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:44.585528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.585438 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-openshift-ca\") pod \"865178fe-5865-488d-9416-452e1b9e2aef\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " Oct 30 15:57:44.585528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.585485 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-xks-ca\") pod \"865178fe-5865-488d-9416-452e1b9e2aef\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " Oct 30 15:57:44.585764 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.585551 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865178fe-5865-488d-9416-452e1b9e2aef-kube-api-access-vf72h\" (UniqueName: \"kubernetes.io/projected/865178fe-5865-488d-9416-452e1b9e2aef-kube-api-access-vf72h\") pod \"865178fe-5865-488d-9416-452e1b9e2aef\" (UID: \"865178fe-5865-488d-9416-452e1b9e2aef\") " Oct 30 15:57:44.585842 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.585791 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-openshift-ca" pod "865178fe-5865-488d-9416-452e1b9e2aef" (UID: "865178fe-5865-488d-9416-452e1b9e2aef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:44.586210 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.586189 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:44.587698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.587670 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-xks-ca" pod "865178fe-5865-488d-9416-452e1b9e2aef" (UID: "865178fe-5865-488d-9416-452e1b9e2aef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:44.587793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.587725 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865178fe-5865-488d-9416-452e1b9e2aef-kube-api-access-vf72h" pod "865178fe-5865-488d-9416-452e1b9e2aef" (UID: "865178fe-5865-488d-9416-452e1b9e2aef"). InnerVolumeSpecName "kube-api-access-vf72h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:44.687190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.687156 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf72h\" (UniqueName: \"kubernetes.io/projected/865178fe-5865-488d-9416-452e1b9e2aef-kube-api-access-vf72h\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:44.687190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:44.687186 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865178fe-5865-488d-9416-452e1b9e2aef-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:45.063269 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.063232 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:45.072231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.072028 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 15:57:45.072231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.072104 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 15:57:45.265849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.260951 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" Oct 30 15:57:45.265849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.265036 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn" event={"ID":"865178fe-5865-488d-9416-452e1b9e2aef","Type":"ContainerDied","Data":"0930b979ba11eefd169ec456bf11497dbc8d1714b9d1a2323b4057549f018ec9"} Oct 30 15:57:45.265849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.265137 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0930b979ba11eefd169ec456bf11497dbc8d1714b9d1a2323b4057549f018ec9" Oct 30 15:57:45.430375 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.430337 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38b9aece-7669-4520-a8e0-5cd3fcea32a5" path="/var/lib/kubelet/pods/38b9aece-7669-4520-a8e0-5cd3fcea32a5/volumes" Oct 30 15:57:45.835846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.835736 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm"] Oct 30 15:57:45.878059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:45.878016 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt"] Oct 30 15:57:46.070849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.065748 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 15:57:46.070849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.065808 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 15:57:46.078407 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.078355 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh"] Oct 30 15:57:46.079597 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079563 4401 cpu_manager.go:414] "containerMap: removing container" podUID="4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" containerName="revoke-keys" Oct 30 15:57:46.079597 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079598 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" containerName="revoke-keys" Oct 30 15:57:46.079800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079641 4401 cpu_manager.go:414] "containerMap: removing container" podUID="865178fe-5865-488d-9416-452e1b9e2aef" containerName="revoke-keys" Oct 30 15:57:46.079800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079650 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865178fe-5865-488d-9416-452e1b9e2aef" containerName="revoke-keys" Oct 30 15:57:46.079800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079696 4401 cpu_manager.go:414] "containerMap: removing container" podUID="26b94c73-08da-4cb8-bd5f-1e5b6eef176f" containerName="revoke-keys" Oct 30 15:57:46.079800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079704 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26b94c73-08da-4cb8-bd5f-1e5b6eef176f" containerName="revoke-keys" Oct 30 15:57:46.080032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079904 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" containerName="revoke-keys" Oct 30 15:57:46.080032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079926 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="865178fe-5865-488d-9416-452e1b9e2aef" containerName="revoke-keys" Oct 30 15:57:46.080032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.079945 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="26b94c73-08da-4cb8-bd5f-1e5b6eef176f" containerName="revoke-keys" Oct 30 15:57:46.089806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.089727 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh"] Oct 30 15:57:46.090025 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.089903 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.108482 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.108453 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2b3f1\"/\"distinct-2-w1-a2b3f1-kserve-self-signed-certs\"" Oct 30 15:57:46.120372 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.120341 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcscx\" (UniqueName: \"kubernetes.io/projected/a87bdf18-4e99-49df-aac9-1d499413de55-kube-api-access-mcscx\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.120557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.120543 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-dshm\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.120728 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.120706 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-tmp-dir\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.120900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.120881 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87bdf18-4e99-49df-aac9-1d499413de55-tls-certs\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.120955 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.120941 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-model-cache\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.121015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.120999 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-home\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.222990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.222953 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-home\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223058 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcscx\" (UniqueName: \"kubernetes.io/projected/a87bdf18-4e99-49df-aac9-1d499413de55-kube-api-access-mcscx\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223089 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-dshm\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223190 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223171 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-tmp-dir\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223339 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223299 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87bdf18-4e99-49df-aac9-1d499413de55-tls-certs\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223399 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223357 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-model-cache\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223394 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-home\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223650 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223620 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-tmp-dir\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.223762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.223697 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-model-cache\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.225401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.225371 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-dshm\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.225786 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.225765 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87bdf18-4e99-49df-aac9-1d499413de55-tls-certs\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.290848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.282020 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcscx\" (UniqueName: \"kubernetes.io/projected/a87bdf18-4e99-49df-aac9-1d499413de55-kube-api-access-mcscx\") pod \"distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.411895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.409465 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:46.738447 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.738348 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" podUID="28621ed5-d455-427d-bedc-bd96556b40f1" containerName="payload-processing" containerID="cri-o://12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5" gracePeriod=25 Oct 30 15:57:46.877301 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:46.877258 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda87bdf18_4e99_49df_aac9_1d499413de55.slice/crio-22169147668a93249f340806aaf4a5d7e8e77d9abea6d2367d967ca3c70e4369 WatchSource:0}: Error finding container 22169147668a93249f340806aaf4a5d7e8e77d9abea6d2367d967ca3c70e4369: Status 404 returned error can't find the container with id 22169147668a93249f340806aaf4a5d7e8e77d9abea6d2367d967ca3c70e4369 Oct 30 15:57:46.879061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.878958 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh"] Oct 30 15:57:46.999687 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:46.999595 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n"] Oct 30 15:57:47.004813 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.004787 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.030487 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.030450 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n"] Oct 30 15:57:47.034616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.033857 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:57:47.034616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.033936 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:57:47.034616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.034022 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:57:47.034616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.034572 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:57:47.043698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.043658 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b"] Oct 30 15:57:47.071911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.053664 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.071911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.053720 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.071911 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.053765 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49blz\" (UniqueName: \"kubernetes.io/projected/e3bd806c-2fa7-43eb-92f8-0997e09a6627-kube-api-access-49blz\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.080557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.080109 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 15:57:47.080557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.080171 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 15:57:47.080557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.080215 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.092079 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.091992 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b"] Oct 30 15:57:47.170905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.160905 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.171109 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.171078 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f47hx\" (UniqueName: \"kubernetes.io/projected/1eb20672-10c1-4da9-99e2-67f3896c83ec-kube-api-access-f47hx\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.171222 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.171202 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.171274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.171255 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.171327 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.171313 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.171472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.171442 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49blz\" (UniqueName: \"kubernetes.io/projected/e3bd806c-2fa7-43eb-92f8-0997e09a6627-kube-api-access-49blz\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.172145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.172121 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.173971 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.173951 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.197120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.197082 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49blz\" (UniqueName: \"kubernetes.io/projected/e3bd806c-2fa7-43eb-92f8-0997e09a6627-kube-api-access-49blz\") pod \"maas-api-revoke-sub-05d12b81-q2w6n\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.272618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.272527 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.272788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.272698 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f47hx\" (UniqueName: \"kubernetes.io/projected/1eb20672-10c1-4da9-99e2-67f3896c83ec-kube-api-access-f47hx\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.272921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.272794 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.273444 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.273412 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.275406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.275385 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.299038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.299002 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f47hx\" (UniqueName: \"kubernetes.io/projected/1eb20672-10c1-4da9-99e2-67f3896c83ec-kube-api-access-f47hx\") pod \"maas-api-revoke-sub-873e56eb-w4x4b\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.343910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.343871 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:47.382455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.382411 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" event={"ID":"a87bdf18-4e99-49df-aac9-1d499413de55","Type":"ContainerStarted","Data":"6671653069cc6dc6c73d6875df5ebe8b32f943438926ad68974e3ec984ade67a"} Oct 30 15:57:47.382630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.382468 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" event={"ID":"a87bdf18-4e99-49df-aac9-1d499413de55","Type":"ContainerStarted","Data":"22169147668a93249f340806aaf4a5d7e8e77d9abea6d2367d967ca3c70e4369"} Oct 30 15:57:47.400515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.400482 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:47.410038 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.410002 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:47.434632 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.434567 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" podStartSLOduration=1.434546439 podStartE2EDuration="1.434546439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:47.430941579 +0000 UTC m=+1176.130214736" watchObservedRunningTime="2026-10-30 15:57:47.434546439 +0000 UTC m=+1176.133819593" Oct 30 15:57:47.788613 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.787922 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n"] Oct 30 15:57:47.976651 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:47.976618 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb20672_10c1_4da9_99e2_67f3896c83ec.slice/crio-94aa8e4a368a2888284b83b233dc628332a6bdd8aeb8b72544f34d8788f6ff13 WatchSource:0}: Error finding container 94aa8e4a368a2888284b83b233dc628332a6bdd8aeb8b72544f34d8788f6ff13: Status 404 returned error can't find the container with id 94aa8e4a368a2888284b83b233dc628332a6bdd8aeb8b72544f34d8788f6ff13 Oct 30 15:57:47.978446 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:47.978390 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b"] Oct 30 15:57:48.070852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.066838 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 15:57:48.070852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.066906 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 15:57:48.308536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.308490 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" containerID="cri-o://8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0" gracePeriod=45 Oct 30 15:57:48.396471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.396427 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" event={"ID":"e3bd806c-2fa7-43eb-92f8-0997e09a6627","Type":"ContainerStarted","Data":"b29d431e87942d465caab007ed11e00ccc2153840594bb52f56bc4e9b95682f2"} Oct 30 15:57:48.396621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.396507 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" event={"ID":"e3bd806c-2fa7-43eb-92f8-0997e09a6627","Type":"ContainerStarted","Data":"78013f6967984fd268f242fca095aefaffceed3c9647566cbce50686852a6d95"} Oct 30 15:57:48.398700 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.398670 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" event={"ID":"1eb20672-10c1-4da9-99e2-67f3896c83ec","Type":"ContainerStarted","Data":"94aa8e4a368a2888284b83b233dc628332a6bdd8aeb8b72544f34d8788f6ff13"} Oct 30 15:57:48.443896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.443841 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" podStartSLOduration=2.443804782 podStartE2EDuration="2.443804782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:48.442169849 +0000 UTC m=+1177.141443007" watchObservedRunningTime="2026-10-30 15:57:48.443804782 +0000 UTC m=+1177.143077938" Oct 30 15:57:48.639153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.639081 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:57:48.719848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.718599 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bd7982-3c33-4b37-9f09-336d50de082d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd7982-3c33-4b37-9f09-336d50de082d-tls-certs\") pod \"a8bd7982-3c33-4b37-9f09-336d50de082d\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " Oct 30 15:57:48.719848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.718673 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-tmp-dir\") pod \"a8bd7982-3c33-4b37-9f09-336d50de082d\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " Oct 30 15:57:48.719848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.718739 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-model-cache\") pod \"a8bd7982-3c33-4b37-9f09-336d50de082d\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " Oct 30 15:57:48.719848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.718880 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bd7982-3c33-4b37-9f09-336d50de082d-kube-api-access-pwjgk\" (UniqueName: \"kubernetes.io/projected/a8bd7982-3c33-4b37-9f09-336d50de082d-kube-api-access-pwjgk\") pod \"a8bd7982-3c33-4b37-9f09-336d50de082d\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " Oct 30 15:57:48.719848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.718990 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-dshm\") pod \"a8bd7982-3c33-4b37-9f09-336d50de082d\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " Oct 30 15:57:48.719848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.719106 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-home\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-home\") pod \"a8bd7982-3c33-4b37-9f09-336d50de082d\" (UID: \"a8bd7982-3c33-4b37-9f09-336d50de082d\") " Oct 30 15:57:48.723568 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.720327 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-model-cache" pod "a8bd7982-3c33-4b37-9f09-336d50de082d" (UID: "a8bd7982-3c33-4b37-9f09-336d50de082d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:48.723568 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.723488 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-tmp-dir" pod "a8bd7982-3c33-4b37-9f09-336d50de082d" (UID: "a8bd7982-3c33-4b37-9f09-336d50de082d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:48.723836 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.723655 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-home" pod "a8bd7982-3c33-4b37-9f09-336d50de082d" (UID: "a8bd7982-3c33-4b37-9f09-336d50de082d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:48.727672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.727635 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bd7982-3c33-4b37-9f09-336d50de082d-kube-api-access-pwjgk" pod "a8bd7982-3c33-4b37-9f09-336d50de082d" (UID: "a8bd7982-3c33-4b37-9f09-336d50de082d"). InnerVolumeSpecName "kube-api-access-pwjgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:48.741569 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.741524 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bd7982-3c33-4b37-9f09-336d50de082d-tls-certs" pod "a8bd7982-3c33-4b37-9f09-336d50de082d" (UID: "a8bd7982-3c33-4b37-9f09-336d50de082d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:48.742042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.742012 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-dshm" pod "a8bd7982-3c33-4b37-9f09-336d50de082d" (UID: "a8bd7982-3c33-4b37-9f09-336d50de082d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:48.821528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.821490 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwjgk\" (UniqueName: \"kubernetes.io/projected/a8bd7982-3c33-4b37-9f09-336d50de082d-kube-api-access-pwjgk\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:48.821528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.821523 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:48.821528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.821534 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:48.821762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.821542 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8bd7982-3c33-4b37-9f09-336d50de082d-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:48.821762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.821553 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:48.821762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:48.821560 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8bd7982-3c33-4b37-9f09-336d50de082d-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:49.055750 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.055265 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 15:57:49.055750 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.055330 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 15:57:49.311076 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.311034 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:57:49.409464 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.409419 4401 generic.go:356] "Generic (PLEG): container finished" podID="e3bd806c-2fa7-43eb-92f8-0997e09a6627" containerID="b29d431e87942d465caab007ed11e00ccc2153840594bb52f56bc4e9b95682f2" exitCode=0 Oct 30 15:57:49.409661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.409489 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" event={"ID":"e3bd806c-2fa7-43eb-92f8-0997e09a6627","Type":"ContainerDied","Data":"b29d431e87942d465caab007ed11e00ccc2153840594bb52f56bc4e9b95682f2"} Oct 30 15:57:49.411175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.411149 4401 generic.go:356] "Generic (PLEG): container finished" podID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerID="8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0" exitCode=0 Oct 30 15:57:49.411351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.411187 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" Oct 30 15:57:49.411351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.411230 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" event={"ID":"a8bd7982-3c33-4b37-9f09-336d50de082d","Type":"ContainerDied","Data":"8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0"} Oct 30 15:57:49.411351 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.411344 4401 scope.go:122] "RemoveContainer" containerID="8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0" Oct 30 15:57:49.411652 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.411630 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj" event={"ID":"a8bd7982-3c33-4b37-9f09-336d50de082d","Type":"ContainerDied","Data":"e73ecbf25eac3ac3c4c76b3c706d73c27ea5cb2131f62a5309b36b7c726c492d"} Oct 30 15:57:49.417534 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.417504 4401 generic.go:356] "Generic (PLEG): container finished" podID="1eb20672-10c1-4da9-99e2-67f3896c83ec" containerID="92970131543bb6906028954a1e151d58c0593a483bdf92c79864cbdf116eab28" exitCode=0 Oct 30 15:57:49.417663 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.417618 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" event={"ID":"1eb20672-10c1-4da9-99e2-67f3896c83ec","Type":"ContainerDied","Data":"92970131543bb6906028954a1e151d58c0593a483bdf92c79864cbdf116eab28"} Oct 30 15:57:49.445575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.445543 4401 scope.go:122] "RemoveContainer" containerID="8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0" Oct 30 15:57:49.476519 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:57:49.474193 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0\": container with ID starting with 8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0 not found: ID does not exist" containerID="8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0" Oct 30 15:57:49.476519 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.474253 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0"} err="failed to get container status \"8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0\": rpc error: code = NotFound desc = could not find container \"8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0\": container with ID starting with 8ebe8bdd7ab88d4a60ea1cf2bb57475554e1419b109c0a20c3d96e6e93fb43f0 not found: ID does not exist" Oct 30 15:57:49.560294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.560186 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj"] Oct 30 15:57:49.566739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:49.566698 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b63c3c/premium-w0-b63c3c-kserve-758657f9cd-hhnmj"] Oct 30 15:57:50.052896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:50.052190 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 15:57:50.052896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:50.052244 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 15:57:50.119053 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:50.119010 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" podUID="9ab32413-1c61-458b-87f9-4696ae69aba9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:57:50.119247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:50.119215 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" podUID="c28756ca-98c8-47ec-a504-689b12e1052e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:57:51.335850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:51.335465 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" containerID="cri-o://97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae" gracePeriod=45 Oct 30 15:57:51.461909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:51.453815 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:51.730011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:51.725465 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:57:51.807851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:51.805647 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:57:51.807851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:51.805727 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:51.815238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:51.815169 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 30 15:57:52.011919 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.007938 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 15:57:52.061977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.061923 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:52.167852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.088156 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:57:52.167852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.088224 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:52.171111 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.171073 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:57:52.174850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.171315 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:52.258698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.258640 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": context deadline exceeded" Oct 30 15:57:52.274810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.274183 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" path="/var/lib/kubelet/pods/a8bd7982-3c33-4b37-9f09-336d50de082d/volumes" Oct 30 15:57:52.399012 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.395088 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" containerID="cri-o://f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37" gracePeriod=44 Oct 30 15:57:52.641882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.613916 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt"] Oct 30 15:57:52.641882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.614234 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": EOF" Oct 30 15:57:52.641882 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:57:52.630176 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:57:52.862414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.674979 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:52.862414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.792063 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:52.862414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.820433 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:57:52.862414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.821525 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 30 15:57:52.862414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.828138 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": dial tcp 10.133.0.91:8000: connect: connection refused" Oct 30 15:57:52.862414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.828299 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:57:52.948490 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.948442 4401 patch_prober.go:29] interesting pod/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:57:52.948746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:52.948714 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:53.075849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.064728 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\") pod \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " Oct 30 15:57:53.075849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.064928 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3bd806c-2fa7-43eb-92f8-0997e09a6627-kube-api-access-49blz\" (UniqueName: \"kubernetes.io/projected/e3bd806c-2fa7-43eb-92f8-0997e09a6627-kube-api-access-49blz\") pod \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " Oct 30 15:57:53.075849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.065160 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-xks-ca\") pod \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " Oct 30 15:57:53.075849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.069232 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:53.075849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.074958 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:57:53.102856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.101966 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 15:57:53.102856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.102694 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:53.155250 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.155155 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:53.164853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.156790 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:53.179858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.178650 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:57:53.206915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.206038 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:53.266592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.263938 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:53.266592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.264520 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:57:53.312537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.309610 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca" pod "e3bd806c-2fa7-43eb-92f8-0997e09a6627" (UID: "e3bd806c-2fa7-43eb-92f8-0997e09a6627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:53.315349 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:53.315323 4401 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/e3bd806c-2fa7-43eb-92f8-0997e09a6627/volumes/kubernetes.io~configmap/maas-api-openshift-ca Oct 30 15:57:53.315573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.315548 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca" pod "e3bd806c-2fa7-43eb-92f8-0997e09a6627" (UID: "e3bd806c-2fa7-43eb-92f8-0997e09a6627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:53.335995 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.329281 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3bd806c-2fa7-43eb-92f8-0997e09a6627-kube-api-access-49blz" pod "e3bd806c-2fa7-43eb-92f8-0997e09a6627" (UID: "e3bd806c-2fa7-43eb-92f8-0997e09a6627"). InnerVolumeSpecName "kube-api-access-49blz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:53.336479 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.336427 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-xks-ca" pod "e3bd806c-2fa7-43eb-92f8-0997e09a6627" (UID: "e3bd806c-2fa7-43eb-92f8-0997e09a6627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:53.345460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.345356 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:53.615247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.315039 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\") pod \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\" (UID: \"e3bd806c-2fa7-43eb-92f8-0997e09a6627\") " Oct 30 15:57:53.749921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.749817 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:53.751998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.751960 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3bd806c-2fa7-43eb-92f8-0997e09a6627-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:53.752565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.752244 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49blz\" (UniqueName: \"kubernetes.io/projected/e3bd806c-2fa7-43eb-92f8-0997e09a6627-kube-api-access-49blz\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:53.786817 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.777331 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": dial tcp 10.133.0.91:8000: connect: connection refused" Oct 30 15:57:53.788418 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:53.786583 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 30 15:57:54.034081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.026135 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb20672-10c1-4da9-99e2-67f3896c83ec-kube-api-access-f47hx\" (UniqueName: \"kubernetes.io/projected/1eb20672-10c1-4da9-99e2-67f3896c83ec-kube-api-access-f47hx\") pod \"1eb20672-10c1-4da9-99e2-67f3896c83ec\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " Oct 30 15:57:54.034081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.026269 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-xks-ca\") pod \"1eb20672-10c1-4da9-99e2-67f3896c83ec\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " Oct 30 15:57:54.034081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.026587 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-openshift-ca\") pod \"1eb20672-10c1-4da9-99e2-67f3896c83ec\" (UID: \"1eb20672-10c1-4da9-99e2-67f3896c83ec\") " Oct 30 15:57:54.083718 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:57:54.083673 4401 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ca1b88e_6358_49af_bcc8_1eba20dd115a.slice/crio-conmon-97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:57:54.115367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.115314 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-openshift-ca" pod "1eb20672-10c1-4da9-99e2-67f3896c83ec" (UID: "1eb20672-10c1-4da9-99e2-67f3896c83ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:57:54.116761 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.097810 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.118359 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb20672-10c1-4da9-99e2-67f3896c83ec-kube-api-access-f47hx" pod "1eb20672-10c1-4da9-99e2-67f3896c83ec" (UID: "1eb20672-10c1-4da9-99e2-67f3896c83ec"). InnerVolumeSpecName "kube-api-access-f47hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.120018 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.125482 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.131266 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.134397 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.146255 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b" event={"ID":"1eb20672-10c1-4da9-99e2-67f3896c83ec","Type":"ContainerDied","Data":"94aa8e4a368a2888284b83b233dc628332a6bdd8aeb8b72544f34d8788f6ff13"} Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.169533 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94aa8e4a368a2888284b83b233dc628332a6bdd8aeb8b72544f34d8788f6ff13" Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.169557 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n" event={"ID":"e3bd806c-2fa7-43eb-92f8-0997e09a6627","Type":"ContainerDied","Data":"78013f6967984fd268f242fca095aefaffceed3c9647566cbce50686852a6d95"} Oct 30 15:57:54.169630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.169589 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78013f6967984fd268f242fca095aefaffceed3c9647566cbce50686852a6d95" Oct 30 15:57:54.177259 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.176869 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:57:54.212059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.203416 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:57:54.212059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.207525 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-xks-ca" pod "1eb20672-10c1-4da9-99e2-67f3896c83ec" (UID: "1eb20672-10c1-4da9-99e2-67f3896c83ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:54.212059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.207638 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:57:54.212059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.208216 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:57:54.251059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.249240 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:57:54.267993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.267896 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54"] Oct 30 15:57:54.273845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.270332 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f47hx\" (UniqueName: \"kubernetes.io/projected/1eb20672-10c1-4da9-99e2-67f3896c83ec-kube-api-access-f47hx\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.273845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.270370 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1eb20672-10c1-4da9-99e2-67f3896c83ec-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.273845 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.273162 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60954d45-cbm54"] Oct 30 15:57:54.299804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.297300 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:57:54.517424 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.517386 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:57:54.576580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.576548 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ca1b88e-6358-49af-bcc8-1eba20dd115a-kube-api-access-cg7px\" (UniqueName: \"kubernetes.io/projected/0ca1b88e-6358-49af-bcc8-1eba20dd115a-kube-api-access-cg7px\") pod \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " Oct 30 15:57:54.576753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.576617 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-model-cache\") pod \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " Oct 30 15:57:54.576753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.576710 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-home\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-home\") pod \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " Oct 30 15:57:54.576753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.576738 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tls-certs\") pod \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " Oct 30 15:57:54.576960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.576866 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tmp-dir\") pod \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " Oct 30 15:57:54.576960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.576917 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-dshm\") pod \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\" (UID: \"0ca1b88e-6358-49af-bcc8-1eba20dd115a\") " Oct 30 15:57:54.577993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.577757 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-home" pod "0ca1b88e-6358-49af-bcc8-1eba20dd115a" (UID: "0ca1b88e-6358-49af-bcc8-1eba20dd115a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.578357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.578127 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-model-cache" pod "0ca1b88e-6358-49af-bcc8-1eba20dd115a" (UID: "0ca1b88e-6358-49af-bcc8-1eba20dd115a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.578788 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.578766 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tmp-dir" pod "0ca1b88e-6358-49af-bcc8-1eba20dd115a" (UID: "0ca1b88e-6358-49af-bcc8-1eba20dd115a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.580307 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.580279 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca1b88e-6358-49af-bcc8-1eba20dd115a-kube-api-access-cg7px" pod "0ca1b88e-6358-49af-bcc8-1eba20dd115a" (UID: "0ca1b88e-6358-49af-bcc8-1eba20dd115a"). InnerVolumeSpecName "kube-api-access-cg7px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:54.580417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.580281 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-dshm" pod "0ca1b88e-6358-49af-bcc8-1eba20dd115a" (UID: "0ca1b88e-6358-49af-bcc8-1eba20dd115a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.580809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.580708 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tls-certs" pod "0ca1b88e-6358-49af-bcc8-1eba20dd115a" (UID: "0ca1b88e-6358-49af-bcc8-1eba20dd115a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:54.678361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.678324 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.678361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.678357 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.678361 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.678365 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.678838 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.678373 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.678838 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.678382 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg7px\" (UniqueName: \"kubernetes.io/projected/0ca1b88e-6358-49af-bcc8-1eba20dd115a-kube-api-access-cg7px\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.678838 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.678393 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ca1b88e-6358-49af-bcc8-1eba20dd115a-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.718428 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.718392 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:57:54.779097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779010 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-dshm\") pod \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " Oct 30 15:57:54.779097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779059 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tls-certs\") pod \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " Oct 30 15:57:54.779332 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779102 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-model-cache\") pod \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " Oct 30 15:57:54.779332 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779161 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-home\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-home\") pod \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " Oct 30 15:57:54.779332 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779246 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tmp-dir\") pod \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " Oct 30 15:57:54.779541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779472 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-model-cache" pod "6dba1878-532c-4b2c-ac0f-b9a22c88410b" (UID: "6dba1878-532c-4b2c-ac0f-b9a22c88410b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.779541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779486 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-home" pod "6dba1878-532c-4b2c-ac0f-b9a22c88410b" (UID: "6dba1878-532c-4b2c-ac0f-b9a22c88410b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.779737 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779709 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tmp-dir" pod "6dba1878-532c-4b2c-ac0f-b9a22c88410b" (UID: "6dba1878-532c-4b2c-ac0f-b9a22c88410b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.779872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779810 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.779872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779839 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.779872 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.779855 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.781435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.781410 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tls-certs" pod "6dba1878-532c-4b2c-ac0f-b9a22c88410b" (UID: "6dba1878-532c-4b2c-ac0f-b9a22c88410b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:57:54.783465 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.783439 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-dshm" pod "6dba1878-532c-4b2c-ac0f-b9a22c88410b" (UID: "6dba1878-532c-4b2c-ac0f-b9a22c88410b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:57:54.888401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.888323 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dba1878-532c-4b2c-ac0f-b9a22c88410b-kube-api-access-82qws\" (UniqueName: \"kubernetes.io/projected/6dba1878-532c-4b2c-ac0f-b9a22c88410b-kube-api-access-82qws\") pod \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\" (UID: \"6dba1878-532c-4b2c-ac0f-b9a22c88410b\") " Oct 30 15:57:54.889220 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.889189 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dba1878-532c-4b2c-ac0f-b9a22c88410b-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.889368 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.889221 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dba1878-532c-4b2c-ac0f-b9a22c88410b-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:54.891057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.891027 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dba1878-532c-4b2c-ac0f-b9a22c88410b-kube-api-access-82qws" pod "6dba1878-532c-4b2c-ac0f-b9a22c88410b" (UID: "6dba1878-532c-4b2c-ac0f-b9a22c88410b"). InnerVolumeSpecName "kube-api-access-82qws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:57:54.990433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:54.990393 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82qws\" (UniqueName: \"kubernetes.io/projected/6dba1878-532c-4b2c-ac0f-b9a22c88410b-kube-api-access-82qws\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:57:55.214625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.205464 4401 generic.go:356] "Generic (PLEG): container finished" podID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerID="f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37" exitCode=0 Oct 30 15:57:55.214625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.205574 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" event={"ID":"6dba1878-532c-4b2c-ac0f-b9a22c88410b","Type":"ContainerDied","Data":"f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37"} Oct 30 15:57:55.214625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.205610 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" event={"ID":"6dba1878-532c-4b2c-ac0f-b9a22c88410b","Type":"ContainerDied","Data":"98e5accffab820ec7e80481e13caf7aae25caa08d379c0bd871771d7ccc63c16"} Oct 30 15:57:55.214625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.205636 4401 scope.go:122] "RemoveContainer" containerID="f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37" Oct 30 15:57:55.214625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.205764 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" Oct 30 15:57:55.242146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.239743 4401 generic.go:356] "Generic (PLEG): container finished" podID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerID="97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae" exitCode=0 Oct 30 15:57:55.246083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.243451 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" Oct 30 15:57:55.266950 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:57:55.266121 4401 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dba1878_532c_4b2c_ac0f_b9a22c88410b.slice/crio-98e5accffab820ec7e80481e13caf7aae25caa08d379c0bd871771d7ccc63c16\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6dba1878_532c_4b2c_ac0f_b9a22c88410b.slice\": RecentStats: unable to find data in memory cache]" Oct 30 15:57:55.275581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.273881 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" event={"ID":"0ca1b88e-6358-49af-bcc8-1eba20dd115a","Type":"ContainerDied","Data":"97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae"} Oct 30 15:57:55.275581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.273944 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj" event={"ID":"0ca1b88e-6358-49af-bcc8-1eba20dd115a","Type":"ContainerDied","Data":"d2db846026e174793db5a27a3b771f32272012955a658f23e8618e226c3d1f10"} Oct 30 15:57:55.356381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.356348 4401 scope.go:122] "RemoveContainer" containerID="f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37" Oct 30 15:57:55.360022 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:57:55.357388 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37\": container with ID starting with f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37 not found: ID does not exist" containerID="f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37" Oct 30 15:57:55.360022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.357435 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37"} err="failed to get container status \"f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37\": rpc error: code = NotFound desc = could not find container \"f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37\": container with ID starting with f6d13bd9f02f95d193a600453d09e1d40117df73f0806ffe6514cfbb8e0bef37 not found: ID does not exist" Oct 30 15:57:55.360022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.357460 4401 scope.go:122] "RemoveContainer" containerID="97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae" Oct 30 15:57:55.406129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.406080 4401 scope.go:122] "RemoveContainer" containerID="97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae" Oct 30 15:57:55.407370 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:57:55.407329 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae\": container with ID starting with 97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae not found: ID does not exist" containerID="97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae" Oct 30 15:57:55.407507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.407382 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae"} err="failed to get container status \"97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae\": rpc error: code = NotFound desc = could not find container \"97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae\": container with ID starting with 97599272729ee5b817b194035dd78ffbd7319fa471f23332424bd32ec822d4ae not found: ID does not exist" Oct 30 15:57:55.640023 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.639962 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edaaaab-ad02-4621-b9ad-83a2196fc572" path="/var/lib/kubelet/pods/6edaaaab-ad02-4621-b9ad-83a2196fc572/volumes" Oct 30 15:57:55.641076 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.641044 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79"] Oct 30 15:57:55.641265 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.641250 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79"] Oct 30 15:57:55.706971 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.706876 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-c129d1/premium-w6-c129d1-kserve-5f999d9bf8-fvr79" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 15:57:55.723436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.719874 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj"] Oct 30 15:57:55.726470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.726436 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-c129d1/distinct-w6-c129d1-kserve-5dd8746b9c-mw6zj"] Oct 30 15:57:55.802747 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.802705 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc"] Oct 30 15:57:55.808999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:55.808970 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-b63c3c-t24sc"] Oct 30 15:57:56.036195 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:56.036105 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb"] Oct 30 15:57:56.042418 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:56.042386 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns"] Oct 30 15:57:56.049093 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:56.049051 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f692dfd1-6g9fb"] Oct 30 15:57:56.055515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:56.055480 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-230431fe-rzjns"] Oct 30 15:57:56.186217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:56.186176 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" podUID="28621ed5-d455-427d-bedc-bd96556b40f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:57:56.410246 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:56.410210 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:56.422932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:56.422903 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:57.277223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.277173 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:57:57.435618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.435575 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" path="/var/lib/kubelet/pods/0ca1b88e-6358-49af-bcc8-1eba20dd115a/volumes" Oct 30 15:57:57.436324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.436298 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="258a70c7-fa54-4eef-82fb-6b893604b585" path="/var/lib/kubelet/pods/258a70c7-fa54-4eef-82fb-6b893604b585/volumes" Oct 30 15:57:57.436980 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.436956 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" path="/var/lib/kubelet/pods/6dba1878-532c-4b2c-ac0f-b9a22c88410b/volumes" Oct 30 15:57:57.438080 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.438054 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98153aa6-6b69-4e94-b9ff-e8527e72f819" path="/var/lib/kubelet/pods/98153aa6-6b69-4e94-b9ff-e8527e72f819/volumes" Oct 30 15:57:57.438724 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.438703 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5" path="/var/lib/kubelet/pods/c4e8f38f-9de8-493a-a1d0-2d0e1bf1a0e5/volumes" Oct 30 15:57:57.440972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.440942 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4"] Oct 30 15:57:57.442025 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.441998 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" Oct 30 15:57:57.442025 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442024 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442060 4401 cpu_manager.go:414] "containerMap: removing container" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442068 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442093 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e3bd806c-2fa7-43eb-92f8-0997e09a6627" containerName="revoke-keys" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442102 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3bd806c-2fa7-43eb-92f8-0997e09a6627" containerName="revoke-keys" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442116 4401 cpu_manager.go:414] "containerMap: removing container" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442121 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442137 4401 cpu_manager.go:414] "containerMap: removing container" podUID="1eb20672-10c1-4da9-99e2-67f3896c83ec" containerName="revoke-keys" Oct 30 15:57:57.442185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442144 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb20672-10c1-4da9-99e2-67f3896c83ec" containerName="revoke-keys" Oct 30 15:57:57.442468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442287 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ca1b88e-6358-49af-bcc8-1eba20dd115a" containerName="main" Oct 30 15:57:57.442468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442305 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3bd806c-2fa7-43eb-92f8-0997e09a6627" containerName="revoke-keys" Oct 30 15:57:57.442468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442314 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dba1878-532c-4b2c-ac0f-b9a22c88410b" containerName="main" Oct 30 15:57:57.442468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442329 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb20672-10c1-4da9-99e2-67f3896c83ec" containerName="revoke-keys" Oct 30 15:57:57.442468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.442343 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bd7982-3c33-4b37-9f09-336d50de082d" containerName="main" Oct 30 15:57:57.448555 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.448534 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.463731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.463701 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4"] Oct 30 15:57:57.465581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.465541 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-9e2b29\"/\"distinct-2-w2-9e2b29-kserve-self-signed-certs\"" Oct 30 15:57:57.540965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.540873 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cplp\" (UniqueName: \"kubernetes.io/projected/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-kube-api-access-5cplp\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.540965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.540954 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tls-certs\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.541158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.541017 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-dshm\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.541158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.541122 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tmp-dir\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.541254 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.541237 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-model-cache\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.541290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.541265 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-home\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.653852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.650964 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tls-certs\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.653852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.651017 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-dshm\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.653852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.651090 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tmp-dir\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.653852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.651223 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-model-cache\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.653852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.651268 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-home\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.653852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.651490 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cplp\" (UniqueName: \"kubernetes.io/projected/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-kube-api-access-5cplp\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.660848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.659317 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tmp-dir\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.660848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.660416 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-model-cache\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.660848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.660808 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-home\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.680849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.675502 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tls-certs\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.689847 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.689060 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" podUID="0c7ea812-ccc6-44d1-88a9-425edea737e5" containerName="payload-pre-processing" containerID="cri-o://3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff" gracePeriod=25 Oct 30 15:57:57.701861 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.697157 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cplp\" (UniqueName: \"kubernetes.io/projected/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-kube-api-access-5cplp\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.714424 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.706636 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-dshm\") pod \"distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:57.776217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:57.771220 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:58.154567 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:57:58.154535 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda129bf09_98a5_49c7_a89a_a4fda9f17ee3.slice/crio-3dd76c8e3e24b262cb60c61e678106871555dea2500bf30b541c49b93c46abb4 WatchSource:0}: Error finding container 3dd76c8e3e24b262cb60c61e678106871555dea2500bf30b541c49b93c46abb4: Status 404 returned error can't find the container with id 3dd76c8e3e24b262cb60c61e678106871555dea2500bf30b541c49b93c46abb4 Oct 30 15:57:58.158702 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:58.158677 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4"] Oct 30 15:57:58.279999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:58.279950 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" event={"ID":"a129bf09-98a5-49c7-a89a-a4fda9f17ee3","Type":"ContainerStarted","Data":"a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230"} Oct 30 15:57:58.280380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:58.280015 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" event={"ID":"a129bf09-98a5-49c7-a89a-a4fda9f17ee3","Type":"ContainerStarted","Data":"3dd76c8e3e24b262cb60c61e678106871555dea2500bf30b541c49b93c46abb4"} Oct 30 15:57:58.325607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:58.325554 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" podStartSLOduration=1.325539867 podStartE2EDuration="1.325539867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:58.324611263 +0000 UTC m=+1187.023884420" watchObservedRunningTime="2026-10-30 15:57:58.325539867 +0000 UTC m=+1187.024813023" Oct 30 15:57:58.772438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:58.772405 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:57:59.430105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.430074 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp"] Oct 30 15:57:59.436698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.436670 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.442234 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.442200 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp"] Oct 30 15:57:59.455722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.452417 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54"] Oct 30 15:57:59.466400 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.459731 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.469941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.469911 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:57:59.470129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.470000 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:57:59.483011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.482975 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54"] Oct 30 15:57:59.487120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.487094 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:57:59.487292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.487187 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:57:59.489803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.489766 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl9cl\" (UniqueName: \"kubernetes.io/projected/f5d87eb1-e2c5-4886-8345-10368cfff705-kube-api-access-tl9cl\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.489964 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.489879 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.489964 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.489957 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.490089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.490035 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.490089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.490077 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.490319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.490301 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8tm\" (UniqueName: \"kubernetes.io/projected/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-kube-api-access-zb8tm\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.591966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.591924 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl9cl\" (UniqueName: \"kubernetes.io/projected/f5d87eb1-e2c5-4886-8345-10368cfff705-kube-api-access-tl9cl\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.592187 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.592016 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.592187 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.592074 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.592187 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.592134 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.592187 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.592167 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.592405 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.592309 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8tm\" (UniqueName: \"kubernetes.io/projected/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-kube-api-access-zb8tm\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.592886 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.592850 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.593006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.592882 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.594661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.594631 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.594751 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.594712 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.617972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.617935 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8tm\" (UniqueName: \"kubernetes.io/projected/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-kube-api-access-zb8tm\") pod \"maas-api-revoke-sub-9adfa703-x5s54\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:57:59.617972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.617962 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl9cl\" (UniqueName: \"kubernetes.io/projected/f5d87eb1-e2c5-4886-8345-10368cfff705-kube-api-access-tl9cl\") pod \"maas-api-revoke-sub-5b744b60-6nwbp\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.762930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.757095 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:57:59.815257 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:57:59.815209 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:58:00.198356 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.197588 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" podUID="c28756ca-98c8-47ec-a504-689b12e1052e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:58:00.244683 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.244221 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" podUID="9ab32413-1c61-458b-87f9-4696ae69aba9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:58:00.337263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.337214 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx"] Oct 30 15:58:00.349630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.349581 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-c129d1-tzsdx"] Oct 30 15:58:00.498109 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.498069 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp"] Oct 30 15:58:00.591134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.589883 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54"] Oct 30 15:58:00.596284 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:00.596253 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f83c136_d8fd_4d50_bf3a_9a3db8cf9403.slice/crio-187e3dba0151b141e15aad3b54cb3c810c5f77ba7586fbdfa1d7340c7c16290a WatchSource:0}: Error finding container 187e3dba0151b141e15aad3b54cb3c810c5f77ba7586fbdfa1d7340c7c16290a: Status 404 returned error can't find the container with id 187e3dba0151b141e15aad3b54cb3c810c5f77ba7586fbdfa1d7340c7c16290a Oct 30 15:58:00.884714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.884642 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" containerID="cri-o://4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871" gracePeriod=45 Oct 30 15:58:00.900279 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:00.900211 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" containerID="cri-o://03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91" gracePeriod=45 Oct 30 15:58:01.205284 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.205245 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:58:01.232237 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.232203 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:58:01.292363 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292269 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tmp-dir\") pod \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " Oct 30 15:58:01.292363 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292354 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tls-certs\") pod \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " Oct 30 15:58:01.292596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292435 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-home\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-home\") pod \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " Oct 30 15:58:01.292596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292572 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tmp-dir" pod "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" (UID: "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.292751 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292727 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-home" pod "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" (UID: "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.292867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292812 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-kube-api-access-mc5jf\") pod \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " Oct 30 15:58:01.292938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292887 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-model-cache\") pod \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " Oct 30 15:58:01.293001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.292990 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-dshm\") pod \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\" (UID: \"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec\") " Oct 30 15:58:01.293178 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.293134 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-model-cache" pod "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" (UID: "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.294032 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.294002 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.294139 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.294036 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.294139 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.294049 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.296990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.296954 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-dshm" pod "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" (UID: "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.297626 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.297603 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-kube-api-access-mc5jf" pod "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" (UID: "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec"). InnerVolumeSpecName "kube-api-access-mc5jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:01.297744 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.297705 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tls-certs" pod "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" (UID: "c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:01.329498 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.329456 4401 generic.go:356] "Generic (PLEG): container finished" podID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerID="03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91" exitCode=0 Oct 30 15:58:01.329712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.329516 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" Oct 30 15:58:01.329712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.329537 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" event={"ID":"8e596300-e655-4aeb-ac47-feb28b3463dc","Type":"ContainerDied","Data":"03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91"} Oct 30 15:58:01.329712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.329590 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt" event={"ID":"8e596300-e655-4aeb-ac47-feb28b3463dc","Type":"ContainerDied","Data":"e7f220dbfc4189c08846d63172d6c208589e62d69421b4e709ab9e60131e08b8"} Oct 30 15:58:01.329712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.329614 4401 scope.go:122] "RemoveContainer" containerID="03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91" Oct 30 15:58:01.333409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.333102 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" event={"ID":"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403","Type":"ContainerStarted","Data":"af965f06a21cc052c0f18b3bfb07e8d7f1fa6eb0f9f72f5882287ef579197a48"} Oct 30 15:58:01.333409 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.333169 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" event={"ID":"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403","Type":"ContainerStarted","Data":"187e3dba0151b141e15aad3b54cb3c810c5f77ba7586fbdfa1d7340c7c16290a"} Oct 30 15:58:01.335342 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.335311 4401 generic.go:356] "Generic (PLEG): container finished" podID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerID="4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871" exitCode=0 Oct 30 15:58:01.335528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.335417 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" event={"ID":"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec","Type":"ContainerDied","Data":"4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871"} Oct 30 15:58:01.335528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.335456 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" event={"ID":"c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec","Type":"ContainerDied","Data":"d554c6067597e46c8f112763c3a7a3a9502394e41bc3431e8766b112bc08df14"} Oct 30 15:58:01.335528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.335522 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm" Oct 30 15:58:01.346640 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.345602 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" event={"ID":"f5d87eb1-e2c5-4886-8345-10368cfff705","Type":"ContainerStarted","Data":"ddeb44b23e5ac983a541798d7d876ab5f80910dc41de889504c605005e80d931"} Oct 30 15:58:01.346640 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.345675 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" event={"ID":"f5d87eb1-e2c5-4886-8345-10368cfff705","Type":"ContainerStarted","Data":"dd6cdcb27d570aa951b5a2759d184714b5f8db603740bb5b8e228e06d9810a3b"} Oct 30 15:58:01.350259 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.350202 4401 scope.go:122] "RemoveContainer" containerID="03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91" Oct 30 15:58:01.350778 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:01.350610 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91\": container with ID starting with 03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91 not found: ID does not exist" containerID="03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91" Oct 30 15:58:01.350778 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.350653 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91"} err="failed to get container status \"03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91\": rpc error: code = NotFound desc = could not find container \"03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91\": container with ID starting with 03c08389ea0f478b7df03a24bf6f7d01f91e9b8bb6adb17d200c4ce34fe57a91 not found: ID does not exist" Oct 30 15:58:01.350778 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.350678 4401 scope.go:122] "RemoveContainer" containerID="4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871" Oct 30 15:58:01.364327 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.364304 4401 scope.go:122] "RemoveContainer" containerID="4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871" Oct 30 15:58:01.364686 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:01.364661 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871\": container with ID starting with 4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871 not found: ID does not exist" containerID="4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871" Oct 30 15:58:01.364768 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.364695 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871"} err="failed to get container status \"4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871\": rpc error: code = NotFound desc = could not find container \"4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871\": container with ID starting with 4c06399946d2bf51c98480b46d57779de6570491202c4cbd7818b78422106871 not found: ID does not exist" Oct 30 15:58:01.394607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394571 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-tmp-dir\") pod \"8e596300-e655-4aeb-ac47-feb28b3463dc\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " Oct 30 15:58:01.394784 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394672 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-model-cache\") pod \"8e596300-e655-4aeb-ac47-feb28b3463dc\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " Oct 30 15:58:01.394784 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394697 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-dshm\") pod \"8e596300-e655-4aeb-ac47-feb28b3463dc\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " Oct 30 15:58:01.394784 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394730 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e596300-e655-4aeb-ac47-feb28b3463dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e596300-e655-4aeb-ac47-feb28b3463dc-tls-certs\") pod \"8e596300-e655-4aeb-ac47-feb28b3463dc\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " Oct 30 15:58:01.394784 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394758 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e596300-e655-4aeb-ac47-feb28b3463dc-kube-api-access-hb8gm\" (UniqueName: \"kubernetes.io/projected/8e596300-e655-4aeb-ac47-feb28b3463dc-kube-api-access-hb8gm\") pod \"8e596300-e655-4aeb-ac47-feb28b3463dc\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " Oct 30 15:58:01.395096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394893 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-tmp-dir" pod "8e596300-e655-4aeb-ac47-feb28b3463dc" (UID: "8e596300-e655-4aeb-ac47-feb28b3463dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.395096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394957 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-home\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-home\") pod \"8e596300-e655-4aeb-ac47-feb28b3463dc\" (UID: \"8e596300-e655-4aeb-ac47-feb28b3463dc\") " Oct 30 15:58:01.395096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.394983 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-model-cache" pod "8e596300-e655-4aeb-ac47-feb28b3463dc" (UID: "8e596300-e655-4aeb-ac47-feb28b3463dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.395268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.395242 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-home" pod "8e596300-e655-4aeb-ac47-feb28b3463dc" (UID: "8e596300-e655-4aeb-ac47-feb28b3463dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.395653 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.395628 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.395749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.395655 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.395749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.395670 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.395749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.395684 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc5jf\" (UniqueName: \"kubernetes.io/projected/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-kube-api-access-mc5jf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.395749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.395698 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.395749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.395711 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.397168 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.397131 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e596300-e655-4aeb-ac47-feb28b3463dc-tls-certs" pod "8e596300-e655-4aeb-ac47-feb28b3463dc" (UID: "8e596300-e655-4aeb-ac47-feb28b3463dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:01.397250 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.397171 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e596300-e655-4aeb-ac47-feb28b3463dc-kube-api-access-hb8gm" pod "8e596300-e655-4aeb-ac47-feb28b3463dc" (UID: "8e596300-e655-4aeb-ac47-feb28b3463dc"). InnerVolumeSpecName "kube-api-access-hb8gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:01.397382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.397363 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-dshm" pod "8e596300-e655-4aeb-ac47-feb28b3463dc" (UID: "8e596300-e655-4aeb-ac47-feb28b3463dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:01.425921 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.425195 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" podStartSLOduration=2.4251761419999998 podStartE2EDuration="2.425176142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:01.407259969 +0000 UTC m=+1190.106533127" watchObservedRunningTime="2026-10-30 15:58:01.425176142 +0000 UTC m=+1190.124449303" Oct 30 15:58:01.434149 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.434112 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f78194cb-cbae-4084-bf47-8d496b6028ff" path="/var/lib/kubelet/pods/f78194cb-cbae-4084-bf47-8d496b6028ff/volumes" Oct 30 15:58:01.452178 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.452114 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" podStartSLOduration=2.452094905 podStartE2EDuration="2.452094905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:01.45118545 +0000 UTC m=+1190.150458608" watchObservedRunningTime="2026-10-30 15:58:01.452094905 +0000 UTC m=+1190.151368065" Oct 30 15:58:01.497617 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.497540 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e596300-e655-4aeb-ac47-feb28b3463dc-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.497617 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.497575 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e596300-e655-4aeb-ac47-feb28b3463dc-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.497617 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.497591 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb8gm\" (UniqueName: \"kubernetes.io/projected/8e596300-e655-4aeb-ac47-feb28b3463dc-kube-api-access-hb8gm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:01.523211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.523165 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm"] Oct 30 15:58:01.529374 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.529340 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/distinct-w4-b24e7b-kserve-c6db7cff9-7bdsm"] Oct 30 15:58:01.681391 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.681358 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt"] Oct 30 15:58:01.687832 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:01.687789 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b24e7b/simulator-w4-b24e7b-kserve-54d647b657-g65lt"] Oct 30 15:58:02.357840 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:02.357794 4401 generic.go:356] "Generic (PLEG): container finished" podID="1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" containerID="af965f06a21cc052c0f18b3bfb07e8d7f1fa6eb0f9f72f5882287ef579197a48" exitCode=0 Oct 30 15:58:02.358041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:02.357883 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" event={"ID":"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403","Type":"ContainerDied","Data":"af965f06a21cc052c0f18b3bfb07e8d7f1fa6eb0f9f72f5882287ef579197a48"} Oct 30 15:58:02.360457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:02.360430 4401 generic.go:356] "Generic (PLEG): container finished" podID="f5d87eb1-e2c5-4886-8345-10368cfff705" containerID="ddeb44b23e5ac983a541798d7d876ab5f80910dc41de889504c605005e80d931" exitCode=0 Oct 30 15:58:02.360585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:02.360509 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" event={"ID":"f5d87eb1-e2c5-4886-8345-10368cfff705","Type":"ContainerDied","Data":"ddeb44b23e5ac983a541798d7d876ab5f80910dc41de889504c605005e80d931"} Oct 30 15:58:03.429850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.429803 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" path="/var/lib/kubelet/pods/8e596300-e655-4aeb-ac47-feb28b3463dc/volumes" Oct 30 15:58:03.430599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.430572 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" path="/var/lib/kubelet/pods/c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec/volumes" Oct 30 15:58:03.571011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.569616 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6"] Oct 30 15:58:03.571011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.569897 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" containerID="cri-o://4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4" gracePeriod=30 Oct 30 15:58:03.581659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.581602 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6"] Oct 30 15:58:03.582006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.581926 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" containerID="cri-o://81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd" gracePeriod=30 Oct 30 15:58:03.600159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.600122 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7"] Oct 30 15:58:03.600449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.600419 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" containerID="cri-o://48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd" gracePeriod=30 Oct 30 15:58:03.613569 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.613538 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:58:03.616986 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.616960 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.696628 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d87eb1-e2c5-4886-8345-10368cfff705-kube-api-access-tl9cl\" (UniqueName: \"kubernetes.io/projected/f5d87eb1-e2c5-4886-8345-10368cfff705-kube-api-access-tl9cl\") pod \"f5d87eb1-e2c5-4886-8345-10368cfff705\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.697412 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-xks-ca\") pod \"f5d87eb1-e2c5-4886-8345-10368cfff705\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.697530 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-openshift-ca\") pod \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.697566 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-xks-ca\") pod \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.697632 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-openshift-ca\") pod \"f5d87eb1-e2c5-4886-8345-10368cfff705\" (UID: \"f5d87eb1-e2c5-4886-8345-10368cfff705\") " Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.697701 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-kube-api-access-zb8tm\" (UniqueName: \"kubernetes.io/projected/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-kube-api-access-zb8tm\") pod \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\" (UID: \"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403\") " Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.701313 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-openshift-ca" pod "1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" (UID: "1f83c136-d8fd-4d50-bf3a-9a3db8cf9403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:03.709041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.701783 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-openshift-ca" pod "f5d87eb1-e2c5-4886-8345-10368cfff705" (UID: "f5d87eb1-e2c5-4886-8345-10368cfff705"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:03.783863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.783169 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-xks-ca" pod "1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" (UID: "1f83c136-d8fd-4d50-bf3a-9a3db8cf9403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:03.791855 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.784296 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-kube-api-access-zb8tm" pod "1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" (UID: "1f83c136-d8fd-4d50-bf3a-9a3db8cf9403"). InnerVolumeSpecName "kube-api-access-zb8tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:03.804333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.801156 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:03.804333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.801203 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:03.804333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.801219 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:03.804333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.801235 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8tm\" (UniqueName: \"kubernetes.io/projected/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403-kube-api-access-zb8tm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:03.825851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.823990 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-xks-ca" pod "f5d87eb1-e2c5-4886-8345-10368cfff705" (UID: "f5d87eb1-e2c5-4886-8345-10368cfff705"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:03.825851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.824144 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d87eb1-e2c5-4886-8345-10368cfff705-kube-api-access-tl9cl" pod "f5d87eb1-e2c5-4886-8345-10368cfff705" (UID: "f5d87eb1-e2c5-4886-8345-10368cfff705"). InnerVolumeSpecName "kube-api-access-tl9cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:03.944875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.944628 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl9cl\" (UniqueName: \"kubernetes.io/projected/f5d87eb1-e2c5-4886-8345-10368cfff705-kube-api-access-tl9cl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:03.944875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:03.944665 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5d87eb1-e2c5-4886-8345-10368cfff705-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:04.428108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.425242 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:58:04.428108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.425297 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:58:04.433247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.433211 4401 generic.go:356] "Generic (PLEG): container finished" podID="9ab32413-1c61-458b-87f9-4696ae69aba9" containerID="98d12d97b494528caf50602dd633980a7544418710a24241a42e618bfd54ab1c" exitCode=0 Oct 30 15:58:04.433748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.433358 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" event={"ID":"9ab32413-1c61-458b-87f9-4696ae69aba9","Type":"ContainerDied","Data":"98d12d97b494528caf50602dd633980a7544418710a24241a42e618bfd54ab1c"} Oct 30 15:58:04.436559 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.436533 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" Oct 30 15:58:04.437580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.437541 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54" event={"ID":"1f83c136-d8fd-4d50-bf3a-9a3db8cf9403","Type":"ContainerDied","Data":"187e3dba0151b141e15aad3b54cb3c810c5f77ba7586fbdfa1d7340c7c16290a"} Oct 30 15:58:04.437715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.437622 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="187e3dba0151b141e15aad3b54cb3c810c5f77ba7586fbdfa1d7340c7c16290a" Oct 30 15:58:04.441790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.441759 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" event={"ID":"f5d87eb1-e2c5-4886-8345-10368cfff705","Type":"ContainerDied","Data":"dd6cdcb27d570aa951b5a2759d184714b5f8db603740bb5b8e228e06d9810a3b"} Oct 30 15:58:04.441977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.441815 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd6cdcb27d570aa951b5a2759d184714b5f8db603740bb5b8e228e06d9810a3b" Oct 30 15:58:04.441977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.441921 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp" Oct 30 15:58:04.451871 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.451836 4401 generic.go:356] "Generic (PLEG): container finished" podID="c28756ca-98c8-47ec-a504-689b12e1052e" containerID="d1611b097b1bdaea618a5b84be286cbd6b6e1cb117341611b78f3afc02b15311" exitCode=0 Oct 30 15:58:04.452083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.451990 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" event={"ID":"c28756ca-98c8-47ec-a504-689b12e1052e","Type":"ContainerDied","Data":"d1611b097b1bdaea618a5b84be286cbd6b6e1cb117341611b78f3afc02b15311"} Oct 30 15:58:04.454135 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.454067 4401 patch_prober.go:29] interesting pod/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:58:04.454236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.454130 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:58:04.454566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.454544 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:58:04.479772 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.479738 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:58:04.562542 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.562505 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab32413-1c61-458b-87f9-4696ae69aba9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab32413-1c61-458b-87f9-4696ae69aba9-plugins-config-volume\") pod \"9ab32413-1c61-458b-87f9-4696ae69aba9\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " Oct 30 15:58:04.562722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.562602 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c28756ca-98c8-47ec-a504-689b12e1052e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28756ca-98c8-47ec-a504-689b12e1052e-plugins-config-volume\") pod \"c28756ca-98c8-47ec-a504-689b12e1052e\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " Oct 30 15:58:04.562722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.562640 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab32413-1c61-458b-87f9-4696ae69aba9-kube-api-access-ld5p2\" (UniqueName: \"kubernetes.io/projected/9ab32413-1c61-458b-87f9-4696ae69aba9-kube-api-access-ld5p2\") pod \"9ab32413-1c61-458b-87f9-4696ae69aba9\" (UID: \"9ab32413-1c61-458b-87f9-4696ae69aba9\") " Oct 30 15:58:04.562900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.562721 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c28756ca-98c8-47ec-a504-689b12e1052e-kube-api-access-n4s6g\" (UniqueName: \"kubernetes.io/projected/c28756ca-98c8-47ec-a504-689b12e1052e-kube-api-access-n4s6g\") pod \"c28756ca-98c8-47ec-a504-689b12e1052e\" (UID: \"c28756ca-98c8-47ec-a504-689b12e1052e\") " Oct 30 15:58:04.563042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.563012 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab32413-1c61-458b-87f9-4696ae69aba9-plugins-config-volume" pod "9ab32413-1c61-458b-87f9-4696ae69aba9" (UID: "9ab32413-1c61-458b-87f9-4696ae69aba9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:04.563123 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.563012 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c28756ca-98c8-47ec-a504-689b12e1052e-plugins-config-volume" pod "c28756ca-98c8-47ec-a504-689b12e1052e" (UID: "c28756ca-98c8-47ec-a504-689b12e1052e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:04.563415 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.563383 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c28756ca-98c8-47ec-a504-689b12e1052e-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:04.563494 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.563418 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ab32413-1c61-458b-87f9-4696ae69aba9-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:04.565177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.565145 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab32413-1c61-458b-87f9-4696ae69aba9-kube-api-access-ld5p2" pod "9ab32413-1c61-458b-87f9-4696ae69aba9" (UID: "9ab32413-1c61-458b-87f9-4696ae69aba9"). InnerVolumeSpecName "kube-api-access-ld5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:04.565290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.565211 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28756ca-98c8-47ec-a504-689b12e1052e-kube-api-access-n4s6g" pod "c28756ca-98c8-47ec-a504-689b12e1052e" (UID: "c28756ca-98c8-47ec-a504-689b12e1052e"). InnerVolumeSpecName "kube-api-access-n4s6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:04.664550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.664455 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld5p2\" (UniqueName: \"kubernetes.io/projected/9ab32413-1c61-458b-87f9-4696ae69aba9-kube-api-access-ld5p2\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:04.664550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:04.664490 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4s6g\" (UniqueName: \"kubernetes.io/projected/c28756ca-98c8-47ec-a504-689b12e1052e-kube-api-access-n4s6g\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:05.178384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.178336 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" podUID="0c7ea812-ccc6-44d1-88a9-425edea737e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:58:05.463686 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.463652 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" Oct 30 15:58:05.464132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.463682 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb" event={"ID":"9ab32413-1c61-458b-87f9-4696ae69aba9","Type":"ContainerDied","Data":"55e490b9039adc02f0ca6fd7559677c4d739eb1774bdad56981d091991dd4415"} Oct 30 15:58:05.464132 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.463746 4401 scope.go:122] "RemoveContainer" containerID="98d12d97b494528caf50602dd633980a7544418710a24241a42e618bfd54ab1c" Oct 30 15:58:05.465612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.465591 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" Oct 30 15:58:05.465612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.465599 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78" event={"ID":"c28756ca-98c8-47ec-a504-689b12e1052e","Type":"ContainerDied","Data":"f7c16802a7d8dafa45851abb5fcf63b5e0d7cbef2ee2747c94d771a00c923cf3"} Oct 30 15:58:05.475762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.475736 4401 scope.go:122] "RemoveContainer" containerID="d1611b097b1bdaea618a5b84be286cbd6b6e1cb117341611b78f3afc02b15311" Oct 30 15:58:05.527320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.527280 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb"] Oct 30 15:58:05.540659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.540614 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63c3c-57b5dcc5f5-br9bb"] Oct 30 15:58:05.569568 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.569525 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78"] Oct 30 15:58:05.575629 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:05.575598 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63c3c-cd6bf887d-zgd78"] Oct 30 15:58:06.224852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:06.220960 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" podUID="28621ed5-d455-427d-bedc-bd96556b40f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:58:07.026957 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.026928 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:58:07.101406 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.101358 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28621ed5-d455-427d-bedc-bd96556b40f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28621ed5-d455-427d-bedc-bd96556b40f1-plugins-config-volume\") pod \"28621ed5-d455-427d-bedc-bd96556b40f1\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " Oct 30 15:58:07.101601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.101565 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28621ed5-d455-427d-bedc-bd96556b40f1-kube-api-access-8475q\" (UniqueName: \"kubernetes.io/projected/28621ed5-d455-427d-bedc-bd96556b40f1-kube-api-access-8475q\") pod \"28621ed5-d455-427d-bedc-bd96556b40f1\" (UID: \"28621ed5-d455-427d-bedc-bd96556b40f1\") " Oct 30 15:58:07.101804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.101772 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28621ed5-d455-427d-bedc-bd96556b40f1-plugins-config-volume" pod "28621ed5-d455-427d-bedc-bd96556b40f1" (UID: "28621ed5-d455-427d-bedc-bd96556b40f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:07.102266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.102243 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28621ed5-d455-427d-bedc-bd96556b40f1-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:07.103833 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.103791 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28621ed5-d455-427d-bedc-bd96556b40f1-kube-api-access-8475q" pod "28621ed5-d455-427d-bedc-bd96556b40f1" (UID: "28621ed5-d455-427d-bedc-bd96556b40f1"). InnerVolumeSpecName "kube-api-access-8475q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:07.204074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.204018 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8475q\" (UniqueName: \"kubernetes.io/projected/28621ed5-d455-427d-bedc-bd96556b40f1-kube-api-access-8475q\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:07.429604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.429565 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab32413-1c61-458b-87f9-4696ae69aba9" path="/var/lib/kubelet/pods/9ab32413-1c61-458b-87f9-4696ae69aba9/volumes" Oct 30 15:58:07.430108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.430091 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c28756ca-98c8-47ec-a504-689b12e1052e" path="/var/lib/kubelet/pods/c28756ca-98c8-47ec-a504-689b12e1052e/volumes" Oct 30 15:58:07.497749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.497716 4401 generic.go:356] "Generic (PLEG): container finished" podID="28621ed5-d455-427d-bedc-bd96556b40f1" containerID="12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5" exitCode=0 Oct 30 15:58:07.497932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.497775 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" Oct 30 15:58:07.497932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.497844 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" event={"ID":"28621ed5-d455-427d-bedc-bd96556b40f1","Type":"ContainerDied","Data":"12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5"} Oct 30 15:58:07.497932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.497902 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw" event={"ID":"28621ed5-d455-427d-bedc-bd96556b40f1","Type":"ContainerDied","Data":"83c95c749ea66cb49e9ce45b68d458686d70544498e2e12800962cfe0e8126a1"} Oct 30 15:58:07.498085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.497932 4401 scope.go:122] "RemoveContainer" containerID="12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5" Oct 30 15:58:07.510310 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.510274 4401 scope.go:122] "RemoveContainer" containerID="12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5" Oct 30 15:58:07.510654 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:07.510630 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5\": container with ID starting with 12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5 not found: ID does not exist" containerID="12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5" Oct 30 15:58:07.510707 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.510664 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5"} err="failed to get container status \"12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5\": rpc error: code = NotFound desc = could not find container \"12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5\": container with ID starting with 12595186c15dee71bb2ddf685421d6027a2935cc4f65b223d1daa7bdfe4faae5 not found: ID does not exist" Oct 30 15:58:07.559128 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.559091 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw"] Oct 30 15:58:07.572592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.572557 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-c129d1-6fc8c86498-5rndw"] Oct 30 15:58:07.772470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.772381 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:58:07.785286 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.785253 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:58:07.981572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.981537 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2"] Oct 30 15:58:07.982408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982383 4401 cpu_manager.go:414] "containerMap: removing container" podUID="28621ed5-d455-427d-bedc-bd96556b40f1" containerName="payload-processing" Oct 30 15:58:07.982460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982410 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28621ed5-d455-427d-bedc-bd96556b40f1" containerName="payload-processing" Oct 30 15:58:07.982460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982433 4401 cpu_manager.go:414] "containerMap: removing container" podUID="f5d87eb1-e2c5-4886-8345-10368cfff705" containerName="revoke-keys" Oct 30 15:58:07.982460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982439 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d87eb1-e2c5-4886-8345-10368cfff705" containerName="revoke-keys" Oct 30 15:58:07.982460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982454 4401 cpu_manager.go:414] "containerMap: removing container" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982460 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982473 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9ab32413-1c61-458b-87f9-4696ae69aba9" containerName="payload-pre-processing" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982478 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab32413-1c61-458b-87f9-4696ae69aba9" containerName="payload-pre-processing" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982487 4401 cpu_manager.go:414] "containerMap: removing container" podUID="c28756ca-98c8-47ec-a504-689b12e1052e" containerName="payload-processing" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982492 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c28756ca-98c8-47ec-a504-689b12e1052e" containerName="payload-processing" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982502 4401 cpu_manager.go:414] "containerMap: removing container" podUID="1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" containerName="revoke-keys" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982507 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" containerName="revoke-keys" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982534 4401 cpu_manager.go:414] "containerMap: removing container" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" Oct 30 15:58:07.982621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982541 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" Oct 30 15:58:07.982890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982647 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2046f38-9cf6-4b6d-9bbf-cd5077bca9ec" containerName="main" Oct 30 15:58:07.982890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982657 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="28621ed5-d455-427d-bedc-bd96556b40f1" containerName="payload-processing" Oct 30 15:58:07.982890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982670 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d87eb1-e2c5-4886-8345-10368cfff705" containerName="revoke-keys" Oct 30 15:58:07.982890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982679 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e596300-e655-4aeb-ac47-feb28b3463dc" containerName="main" Oct 30 15:58:07.982890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982686 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab32413-1c61-458b-87f9-4696ae69aba9" containerName="payload-pre-processing" Oct 30 15:58:07.982890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982693 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="c28756ca-98c8-47ec-a504-689b12e1052e" containerName="payload-processing" Oct 30 15:58:07.982890 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.982700 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" containerName="revoke-keys" Oct 30 15:58:07.988470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:07.988446 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.014119 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.014090 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2"] Oct 30 15:58:08.014650 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.014616 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-home\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.014780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.014718 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcknw\" (UniqueName: \"kubernetes.io/projected/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-kube-api-access-zcknw\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.014870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.014809 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tls-certs\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.014960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.014924 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-dshm\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.015063 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.014971 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tmp-dir\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.015191 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.015168 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-model-cache\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.018767 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.018742 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ba9db9\"/\"openshift-service-ca.crt\"" Oct 30 15:58:08.018767 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.018752 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ba9db9\"/\"distinct-2-w5-ba9db9-kserve-self-signed-certs\"" Oct 30 15:58:08.018949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.018867 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ba9db9\"/\"default-dockercfg-zrw7s\"" Oct 30 15:58:08.018949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.018919 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ba9db9\"/\"kube-root-ca.crt\"" Oct 30 15:58:08.116472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.116427 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tls-certs\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.116920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.116491 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-dshm\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.116920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.116518 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tmp-dir\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.116920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.116598 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-model-cache\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.116920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.116664 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-home\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.116920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.116720 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcknw\" (UniqueName: \"kubernetes.io/projected/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-kube-api-access-zcknw\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.117171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.117067 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tmp-dir\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.117217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.117201 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-model-cache\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.117249 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.117229 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-home\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.119346 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.119318 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-dshm\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.119557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.119540 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tls-certs\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.144367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.144332 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcknw\" (UniqueName: \"kubernetes.io/projected/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-kube-api-access-zcknw\") pod \"distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.343059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.342724 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:08.383777 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.383686 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb"] Oct 30 15:58:08.384260 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.384227 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" containerID="cri-o://2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935" gracePeriod=30 Oct 30 15:58:08.400582 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.400536 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d"] Oct 30 15:58:08.401624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.400979 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" containerID="cri-o://e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534" gracePeriod=30 Oct 30 15:58:08.414888 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.414790 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5"] Oct 30 15:58:08.415156 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.415079 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" containerID="cri-o://22bd8b511fcd0f8e046b15f2ba50745457a1ed9dc24b2eaa0a92e19aa068ef6a" gracePeriod=30 Oct 30 15:58:08.730364 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:08.729975 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:58:09.137258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.135930 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r"] Oct 30 15:58:09.148525 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.148447 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a22a711e-w985r"] Oct 30 15:58:09.208544 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.208300 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2"] Oct 30 15:58:09.245342 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.245310 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:58:09.252133 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.252103 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:58:09.396225 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.396190 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:58:09.404603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.404559 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-podinfo\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.404808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.404636 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-certs\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.404808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.404704 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-credential-socket\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.404808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.404759 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istiod-ca-cert\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.404808 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.404794 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-socket\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.405070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.404882 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-credential-socket\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.405070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.404924 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-data\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.405070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405022 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-podinfo\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.405070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405053 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-certs\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.405278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405092 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-envoy\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.405278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405134 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-kube-api-access-gcnmb\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-kube-api-access-gcnmb\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.405278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405184 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-token\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-token\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.405278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405194 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-credential-socket" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.405278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405196 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istiod-ca-cert" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:09.405278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405243 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-token\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-token\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.405565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405321 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-data\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.405565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405368 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-kube-api-access-8m9x8\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-kube-api-access-8m9x8\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.405565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405409 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-socket\") pod \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\" (UID: \"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84\") " Oct 30 15:58:09.405565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405502 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-certs" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.405565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405516 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-envoy\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.405818 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.405625 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istiod-ca-cert\") pod \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\" (UID: \"4c97ffa2-d41e-4790-b6dc-34f809bf05f2\") " Oct 30 15:58:09.406263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.406234 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istiod-ca-cert" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:09.406524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.406499 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-data" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.407273 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.407244 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-certs" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.407573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.407550 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-socket" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.407804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.407779 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-credential-socket" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.408047 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.408024 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-socket" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.408913 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.408653 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-data" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.409011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.408981 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409011 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409006 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409022 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409037 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409050 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409062 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409075 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409088 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409100 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.409148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.409114 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.410100 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.410061 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-token" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:09.410241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.410177 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-kube-api-access-gcnmb" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "kube-api-access-gcnmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:09.410331 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.410279 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-podinfo" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:09.411284 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.411241 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-envoy" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.412232 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.412208 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-podinfo" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:09.412355 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.412334 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-envoy" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.412455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.412427 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-kube-api-access-8m9x8" pod "4c97ffa2-d41e-4790-b6dc-34f809bf05f2" (UID: "4c97ffa2-d41e-4790-b6dc-34f809bf05f2"). InnerVolumeSpecName "kube-api-access-8m9x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:09.413684 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.413656 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-token" pod "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" (UID: "d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:09.430129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.430090 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28621ed5-d455-427d-bedc-bd96556b40f1" path="/var/lib/kubelet/pods/28621ed5-d455-427d-bedc-bd96556b40f1/volumes" Oct 30 15:58:09.430679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.430656 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ec4fa9-c902-412a-9b67-4db5cdcc5f92" path="/var/lib/kubelet/pods/b4ec4fa9-c902-412a-9b67-4db5cdcc5f92/volumes" Oct 30 15:58:09.510646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510561 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-certs\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.510646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510614 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istiod-ca-cert\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.510916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510669 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-socket\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.510916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510710 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-kube-api-access-ls95n\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-kube-api-access-ls95n\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.510916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510737 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-token\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-token\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.510916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510796 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-credential-socket\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.511159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510923 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-certs" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.511159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.510970 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-envoy\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.511159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.511029 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-socket" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.511159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.511040 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-data\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.511159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.511116 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istiod-ca-cert" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:09.511416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.511212 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-credential-socket" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.511416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.511125 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-podinfo\") pod \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\" (UID: \"4d62b438-6f71-4ea0-9493-3ce8f382fe21\") " Oct 30 15:58:09.511416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.511334 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-data" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.512037 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512008 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m9x8\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-kube-api-access-8m9x8\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512041 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512059 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512074 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512088 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512103 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512119 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512133 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512146 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512147 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcnmb\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-kube-api-access-gcnmb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512161 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512174 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512187 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c97ffa2-d41e-4790-b6dc-34f809bf05f2-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.512578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.512199 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.513214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.513183 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-envoy" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:09.513214 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.513196 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-token" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:09.513347 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.513232 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-kube-api-access-ls95n" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "kube-api-access-ls95n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:09.513382 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.513353 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-podinfo" pod "4d62b438-6f71-4ea0-9493-3ce8f382fe21" (UID: "4d62b438-6f71-4ea0-9493-3ce8f382fe21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:09.618894 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.618849 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls95n\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-kube-api-access-ls95n\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.619111 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.619095 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.619236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.619224 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.619324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.619314 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d62b438-6f71-4ea0-9493-3ce8f382fe21-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:09.759482 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.759442 4401 generic.go:356] "Generic (PLEG): container finished" podID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerID="4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4" exitCode=0 Oct 30 15:58:09.759929 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.759907 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" Oct 30 15:58:09.765958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.763121 4401 generic.go:356] "Generic (PLEG): container finished" podID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerID="81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd" exitCode=0 Oct 30 15:58:09.766338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.763217 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.763361 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" event={"ID":"4d62b438-6f71-4ea0-9493-3ce8f382fe21","Type":"ContainerDied","Data":"4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.772051 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6" event={"ID":"4d62b438-6f71-4ea0-9493-3ce8f382fe21","Type":"ContainerDied","Data":"36107c4295ec958822fa7c86158482d9993365c3533569f8c85672b2f6d2d53d"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.772101 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" event={"ID":"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84","Type":"ContainerDied","Data":"81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.772132 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6" event={"ID":"d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84","Type":"ContainerDied","Data":"df9f4647498a4323d5414af5b39a6128ee1c26f17009392337f5f77a2092f5a7"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.772154 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" event={"ID":"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf","Type":"ContainerStarted","Data":"742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.772173 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" event={"ID":"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf","Type":"ContainerStarted","Data":"e3a1344a9601bd85d19367efa520fb33f2158d56f27ce392e64d0beabac5d2e4"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.772203 4401 scope.go:122] "RemoveContainer" containerID="4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4" Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.773683 4401 generic.go:356] "Generic (PLEG): container finished" podID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerID="48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd" exitCode=0 Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.773940 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" event={"ID":"4c97ffa2-d41e-4790-b6dc-34f809bf05f2","Type":"ContainerDied","Data":"48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.773997 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" event={"ID":"4c97ffa2-d41e-4790-b6dc-34f809bf05f2","Type":"ContainerDied","Data":"7bd7b57489ff9e140ce114fc50fa52d77fbcb40c0c3df52f0221e62f7e2d53b7"} Oct 30 15:58:09.781276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.774210 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7" Oct 30 15:58:09.803978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.803951 4401 scope.go:122] "RemoveContainer" containerID="4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4" Oct 30 15:58:09.804366 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:09.804331 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4\": container with ID starting with 4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4 not found: ID does not exist" containerID="4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4" Oct 30 15:58:09.804471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.804366 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4"} err="failed to get container status \"4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4\": rpc error: code = NotFound desc = could not find container \"4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4\": container with ID starting with 4881ae4d686960f34431a2bff71248fef6432acb4c4dc2eee102f491bfd816a4 not found: ID does not exist" Oct 30 15:58:09.804471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.804384 4401 scope.go:122] "RemoveContainer" containerID="81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd" Oct 30 15:58:09.817025 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.816998 4401 scope.go:122] "RemoveContainer" containerID="81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd" Oct 30 15:58:09.817420 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:09.817395 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd\": container with ID starting with 81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd not found: ID does not exist" containerID="81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd" Oct 30 15:58:09.817508 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.817435 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd"} err="failed to get container status \"81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd\": rpc error: code = NotFound desc = could not find container \"81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd\": container with ID starting with 81b6974675fbdec8185105e664b4650705be4ac9ac934cbd034b2df48d36fefd not found: ID does not exist" Oct 30 15:58:09.817508 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.817454 4401 scope.go:122] "RemoveContainer" containerID="48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd" Oct 30 15:58:09.830064 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.830044 4401 scope.go:122] "RemoveContainer" containerID="48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd" Oct 30 15:58:09.830379 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:09.830358 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd\": container with ID starting with 48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd not found: ID does not exist" containerID="48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd" Oct 30 15:58:09.830447 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.830391 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd"} err="failed to get container status \"48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd\": rpc error: code = NotFound desc = could not find container \"48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd\": container with ID starting with 48b3d22976d2653a497dee5bed7e8e11c0a60c3afdb1df2a67e64b092f870fdd not found: ID does not exist" Oct 30 15:58:09.852646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.852588 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" podStartSLOduration=2.8525721280000003 podStartE2EDuration="2.852572128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:09.850081892 +0000 UTC m=+1198.549355060" watchObservedRunningTime="2026-10-30 15:58:09.852572128 +0000 UTC m=+1198.551845285" Oct 30 15:58:09.911175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.911142 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6"] Oct 30 15:58:09.923985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.923947 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-7m6s6"] Oct 30 15:58:09.955145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.955109 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7"] Oct 30 15:58:09.960894 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:09.960859 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-bh8g7"] Oct 30 15:58:10.043807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:10.043712 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6"] Oct 30 15:58:10.050233 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:10.050197 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b63c3c-openshift-default-5d7c6c46b5-r77s6"] Oct 30 15:58:10.344118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:10.344029 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:11.134849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:11.127916 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6"] Oct 30 15:58:11.179002 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:11.178965 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5059029-r6qg6"] Oct 30 15:58:11.438248 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:11.438131 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" path="/var/lib/kubelet/pods/4c97ffa2-d41e-4790-b6dc-34f809bf05f2/volumes" Oct 30 15:58:11.439165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:11.439141 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" path="/var/lib/kubelet/pods/4d62b438-6f71-4ea0-9493-3ce8f382fe21/volumes" Oct 30 15:58:11.439598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:11.439584 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9fbe44-0c1b-4ca4-afe8-1aea86842094" path="/var/lib/kubelet/pods/aa9fbe44-0c1b-4ca4-afe8-1aea86842094/volumes" Oct 30 15:58:11.440401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:11.440384 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" path="/var/lib/kubelet/pods/d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84/volumes" Oct 30 15:58:13.038867 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.038811 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj"] Oct 30 15:58:13.052741 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.052706 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk"] Oct 30 15:58:13.058457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.058428 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e3682f-8flbj"] Oct 30 15:58:13.063928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.063899 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c65c7b61-zmvrk"] Oct 30 15:58:13.431451 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.431407 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5e774eb-c937-4454-842d-f83beaca68f2" path="/var/lib/kubelet/pods/d5e774eb-c937-4454-842d-f83beaca68f2/volumes" Oct 30 15:58:13.432207 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.432185 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae5fda4-df24-4926-9ed7-de6b988f6ace" path="/var/lib/kubelet/pods/eae5fda4-df24-4926-9ed7-de6b988f6ace/volumes" Oct 30 15:58:13.761671 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.761642 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:58:13.768229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.768198 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:58:13.846077 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.846038 4401 generic.go:356] "Generic (PLEG): container finished" podID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerID="e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534" exitCode=0 Oct 30 15:58:13.846357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.846128 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" Oct 30 15:58:13.846357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.846130 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" event={"ID":"0f366feb-99b7-4023-bc5d-72a66f8eaefe","Type":"ContainerDied","Data":"e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534"} Oct 30 15:58:13.846357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.846226 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d" event={"ID":"0f366feb-99b7-4023-bc5d-72a66f8eaefe","Type":"ContainerDied","Data":"7be6347aeab0b9c895e145c0ff517b1777ab12e1f672faf6e54bc79b96a7d4a9"} Oct 30 15:58:13.846357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.846260 4401 scope.go:122] "RemoveContainer" containerID="e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534" Oct 30 15:58:13.848656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.848626 4401 generic.go:356] "Generic (PLEG): container finished" podID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerID="2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935" exitCode=0 Oct 30 15:58:13.848876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.848679 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" event={"ID":"e64b4190-cdf8-45e6-83b5-aa6081a311f0","Type":"ContainerDied","Data":"2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935"} Oct 30 15:58:13.849023 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.849006 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" event={"ID":"e64b4190-cdf8-45e6-83b5-aa6081a311f0","Type":"ContainerDied","Data":"97314904364bbcce9ba5297955b0e127daeef5ea6c63dbfc8fc0e3eab6286b4f"} Oct 30 15:58:13.849118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.848692 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb" Oct 30 15:58:13.850605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.850583 4401 generic.go:356] "Generic (PLEG): container finished" podID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerID="22bd8b511fcd0f8e046b15f2ba50745457a1ed9dc24b2eaa0a92e19aa068ef6a" exitCode=0 Oct 30 15:58:13.850701 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.850635 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" event={"ID":"0728c928-7b7d-4ed6-a3df-5e35727c71b0","Type":"ContainerDied","Data":"22bd8b511fcd0f8e046b15f2ba50745457a1ed9dc24b2eaa0a92e19aa068ef6a"} Oct 30 15:58:13.852339 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.852318 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:58:13.863587 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.863557 4401 scope.go:122] "RemoveContainer" containerID="e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534" Oct 30 15:58:13.864007 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:13.863978 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534\": container with ID starting with e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534 not found: ID does not exist" containerID="e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534" Oct 30 15:58:13.864117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.864018 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534"} err="failed to get container status \"e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534\": rpc error: code = NotFound desc = could not find container \"e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534\": container with ID starting with e450b15186eb1770200cda4d15edb349b49feec25089ab5e4e0ee58140da4534 not found: ID does not exist" Oct 30 15:58:13.864117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.864042 4401 scope.go:122] "RemoveContainer" containerID="2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935" Oct 30 15:58:13.878592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.878568 4401 scope.go:122] "RemoveContainer" containerID="2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935" Oct 30 15:58:13.878957 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:13.878931 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935\": container with ID starting with 2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935 not found: ID does not exist" containerID="2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935" Oct 30 15:58:13.879062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.878963 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935"} err="failed to get container status \"2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935\": rpc error: code = NotFound desc = could not find container \"2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935\": container with ID starting with 2d0e48a98cbd945b5f0890ebdbaa652886f31dd1942c6e42fe6c95493d75d935 not found: ID does not exist" Oct 30 15:58:13.906970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.906929 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istiod-ca-cert\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.907144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.906996 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-data\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.907144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907092 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-certs\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.907144 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907136 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-kube-api-access-9ph6p\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-kube-api-access-9ph6p\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907209 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-credential-socket\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907246 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-token\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-token\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907366 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-data" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907420 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-certs" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907634 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-envoy\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907667 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-socket\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907701 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istiod-ca-cert\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907747 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-podinfo\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907888 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-credential-socket" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907925 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-kube-api-access-28qw5\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-kube-api-access-28qw5\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.907966 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-podinfo\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.908001 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-certs\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.908037 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-token\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-token\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.908073 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-credential-socket\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.908288 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-data\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.908325 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-envoy\") pod \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\" (UID: \"0f366feb-99b7-4023-bc5d-72a66f8eaefe\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.908402 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-socket\") pod \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\" (UID: \"e64b4190-cdf8-45e6-83b5-aa6081a311f0\") " Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.908839 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-data" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909052 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-credential-socket" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.909154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909053 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-socket" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909481 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909506 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909521 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909536 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909549 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909562 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.909766 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-certs" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.910263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.910170 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istiod-ca-cert" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:13.910740 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.910474 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-socket" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.911897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.911376 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-envoy" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.912512 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.912452 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-envoy" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:13.912768 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.912735 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-kube-api-access-9ph6p" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "kube-api-access-9ph6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:13.913720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.913682 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-token" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:13.913940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.913801 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istiod-ca-cert" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:13.913940 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.913906 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-token" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:13.914088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.913970 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-kube-api-access-28qw5" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "kube-api-access-28qw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:13.914542 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.914520 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-podinfo" pod "0f366feb-99b7-4023-bc5d-72a66f8eaefe" (UID: "0f366feb-99b7-4023-bc5d-72a66f8eaefe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:13.914713 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:13.914685 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-podinfo" pod "e64b4190-cdf8-45e6-83b5-aa6081a311f0" (UID: "e64b4190-cdf8-45e6-83b5-aa6081a311f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:14.010527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010432 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-podinfo\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.010527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010507 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-data\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.010773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010553 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-token\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-token\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.010773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010580 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-certs\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.010773 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010770 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-socket\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.010939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010809 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-envoy\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.010939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010889 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-certs" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:14.010939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010908 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-data" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:14.010939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010897 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istiod-ca-cert\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.011150 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.010982 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-kube-api-access-sqzb9\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-kube-api-access-sqzb9\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.011150 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011044 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-socket" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:14.011150 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011116 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-credential-socket\") pod \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\" (UID: \"0728c928-7b7d-4ed6-a3df-5e35727c71b0\") " Oct 30 15:58:14.011301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011221 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istiod-ca-cert" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:14.011396 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011354 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-credential-socket" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:14.011675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011657 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28qw5\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-kube-api-access-28qw5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011675 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011686 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011695 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011703 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011711 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011719 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011726 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011735 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ph6p\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-kube-api-access-9ph6p\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011743 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011751 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.011755 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011759 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.012301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011767 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e64b4190-cdf8-45e6-83b5-aa6081a311f0-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.012301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011775 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f366feb-99b7-4023-bc5d-72a66f8eaefe-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.012301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011783 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.012301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011791 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.012301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.011798 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f366feb-99b7-4023-bc5d-72a66f8eaefe-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.013074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.013040 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-envoy" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:14.013170 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.013042 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-kube-api-access-sqzb9" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "kube-api-access-sqzb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:14.013207 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.013161 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-token" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:14.013294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.013277 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-podinfo" pod "0728c928-7b7d-4ed6-a3df-5e35727c71b0" (UID: "0728c928-7b7d-4ed6-a3df-5e35727c71b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:14.113088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.113048 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.113088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.113084 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzb9\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-kube-api-access-sqzb9\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.113088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.113097 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.113572 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.113109 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0728c928-7b7d-4ed6-a3df-5e35727c71b0-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:14.268705 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.268564 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d"] Oct 30 15:58:14.279566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.279528 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-n4v7d"] Oct 30 15:58:14.306096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.306053 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb"] Oct 30 15:58:14.313494 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.313460 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-ts2pb"] Oct 30 15:58:14.894850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.889194 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" event={"ID":"0728c928-7b7d-4ed6-a3df-5e35727c71b0","Type":"ContainerDied","Data":"9c33f35c4a3d40ba88c87d8cf800d6befdfa17ce7f33f5fe3f73af90fad19e60"} Oct 30 15:58:14.894850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.889273 4401 scope.go:122] "RemoveContainer" containerID="22bd8b511fcd0f8e046b15f2ba50745457a1ed9dc24b2eaa0a92e19aa068ef6a" Oct 30 15:58:14.894850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.889415 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5" Oct 30 15:58:15.008850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:14.999277 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5"] Oct 30 15:58:15.012887 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:15.009185 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-c129d1-openshift-default-5b5796b49c-pvzb5"] Oct 30 15:58:15.225896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:15.225781 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" podUID="0c7ea812-ccc6-44d1-88a9-425edea737e5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:58:15.437213 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:15.437166 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" path="/var/lib/kubelet/pods/0728c928-7b7d-4ed6-a3df-5e35727c71b0/volumes" Oct 30 15:58:15.438062 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:15.438034 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" path="/var/lib/kubelet/pods/0f366feb-99b7-4023-bc5d-72a66f8eaefe/volumes" Oct 30 15:58:15.439573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:15.439545 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" path="/var/lib/kubelet/pods/e64b4190-cdf8-45e6-83b5-aa6081a311f0/volumes" Oct 30 15:58:18.016182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.016151 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:58:18.047897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.047864 4401 generic.go:356] "Generic (PLEG): container finished" podID="0c7ea812-ccc6-44d1-88a9-425edea737e5" containerID="3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff" exitCode=0 Oct 30 15:58:18.047897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.047890 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" Oct 30 15:58:18.048118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.047937 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" event={"ID":"0c7ea812-ccc6-44d1-88a9-425edea737e5","Type":"ContainerDied","Data":"3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff"} Oct 30 15:58:18.048118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.048000 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt" event={"ID":"0c7ea812-ccc6-44d1-88a9-425edea737e5","Type":"ContainerDied","Data":"9f3f34a50f53e032c48bc361c00f6f6677e8000eee58dd6c936900b1d6f49546"} Oct 30 15:58:18.048118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.048028 4401 scope.go:122] "RemoveContainer" containerID="3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff" Oct 30 15:58:18.059370 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.059339 4401 scope.go:122] "RemoveContainer" containerID="3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff" Oct 30 15:58:18.059714 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:18.059692 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff\": container with ID starting with 3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff not found: ID does not exist" containerID="3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff" Oct 30 15:58:18.059790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.059725 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff"} err="failed to get container status \"3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff\": rpc error: code = NotFound desc = could not find container \"3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff\": container with ID starting with 3d8a0575ba520a1515801a0fae657f8e1c0e167460846ea5269c8499f0ccf3ff not found: ID does not exist" Oct 30 15:58:18.122982 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.122945 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c7ea812-ccc6-44d1-88a9-425edea737e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7ea812-ccc6-44d1-88a9-425edea737e5-plugins-config-volume\") pod \"0c7ea812-ccc6-44d1-88a9-425edea737e5\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " Oct 30 15:58:18.123177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.123044 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c7ea812-ccc6-44d1-88a9-425edea737e5-kube-api-access-s7nj7\" (UniqueName: \"kubernetes.io/projected/0c7ea812-ccc6-44d1-88a9-425edea737e5-kube-api-access-s7nj7\") pod \"0c7ea812-ccc6-44d1-88a9-425edea737e5\" (UID: \"0c7ea812-ccc6-44d1-88a9-425edea737e5\") " Oct 30 15:58:18.123371 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.123346 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c7ea812-ccc6-44d1-88a9-425edea737e5-plugins-config-volume" pod "0c7ea812-ccc6-44d1-88a9-425edea737e5" (UID: "0c7ea812-ccc6-44d1-88a9-425edea737e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:18.123546 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.123529 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c7ea812-ccc6-44d1-88a9-425edea737e5-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:18.125232 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.125207 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c7ea812-ccc6-44d1-88a9-425edea737e5-kube-api-access-s7nj7" pod "0c7ea812-ccc6-44d1-88a9-425edea737e5" (UID: "0c7ea812-ccc6-44d1-88a9-425edea737e5"). InnerVolumeSpecName "kube-api-access-s7nj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:18.225228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.225187 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nj7\" (UniqueName: \"kubernetes.io/projected/0c7ea812-ccc6-44d1-88a9-425edea737e5-kube-api-access-s7nj7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:18.344036 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.343931 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:18.360624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.360598 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:18.417379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.417341 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt"] Oct 30 15:58:18.442515 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:18.442480 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-b24e7b-779bc75f4c-52hvt"] Oct 30 15:58:19.069430 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.069400 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:19.200111 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.200075 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz"] Oct 30 15:58:19.205909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.205878 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z"] Oct 30 15:58:19.211529 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.211496 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15068d95-frh9z"] Oct 30 15:58:19.217512 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.217482 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b0d063d-vbgsz"] Oct 30 15:58:19.489179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.489142 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c7ea812-ccc6-44d1-88a9-425edea737e5" path="/var/lib/kubelet/pods/0c7ea812-ccc6-44d1-88a9-425edea737e5/volumes" Oct 30 15:58:19.490162 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.490134 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb0b022-90b9-4e1c-b305-347a66137d60" path="/var/lib/kubelet/pods/9bb0b022-90b9-4e1c-b305-347a66137d60/volumes" Oct 30 15:58:19.490905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:19.490777 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74e579a-5a54-4e78-9035-0bd06d600337" path="/var/lib/kubelet/pods/e74e579a-5a54-4e78-9035-0bd06d600337/volumes" Oct 30 15:58:21.969552 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:21.969515 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:58:23.050519 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:23.050485 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67"] Oct 30 15:58:23.064453 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:23.064420 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87"] Oct 30 15:58:23.070675 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:23.070647 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4313ab02-5hs67"] Oct 30 15:58:23.076085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:23.076050 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9c4181b-kgs87"] Oct 30 15:58:23.439379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:23.439343 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945701ed-1b70-46d9-b8a9-ef9605c0f068" path="/var/lib/kubelet/pods/945701ed-1b70-46d9-b8a9-ef9605c0f068/volumes" Oct 30 15:58:23.440328 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:23.440295 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfebbd8-8b40-4915-bac2-73284dea8d20" path="/var/lib/kubelet/pods/adfebbd8-8b40-4915-bac2-73284dea8d20/volumes" Oct 30 15:58:27.261471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.261359 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2"] Oct 30 15:58:27.263362 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.263084 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" containerID="cri-o://07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802" gracePeriod=30 Oct 30 15:58:27.377275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.377232 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn"] Oct 30 15:58:27.385920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.385882 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-b24e7b-td2zn"] Oct 30 15:58:27.489887 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.489770 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865178fe-5865-488d-9416-452e1b9e2aef" path="/var/lib/kubelet/pods/865178fe-5865-488d-9416-452e1b9e2aef/volumes" Oct 30 15:58:27.850603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.850562 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw"] Oct 30 15:58:27.851320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851296 4401 cpu_manager.go:414] "containerMap: removing container" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" Oct 30 15:58:27.851320 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851318 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851334 4401 cpu_manager.go:414] "containerMap: removing container" podUID="0c7ea812-ccc6-44d1-88a9-425edea737e5" containerName="payload-pre-processing" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851340 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c7ea812-ccc6-44d1-88a9-425edea737e5" containerName="payload-pre-processing" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851353 4401 cpu_manager.go:414] "containerMap: removing container" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851359 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851371 4401 cpu_manager.go:414] "containerMap: removing container" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851376 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851388 4401 cpu_manager.go:414] "containerMap: removing container" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851393 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851428 4401 cpu_manager.go:414] "containerMap: removing container" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851433 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851440 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851445 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" Oct 30 15:58:27.851526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851530 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c97ffa2-d41e-4790-b6dc-34f809bf05f2" containerName="istio-proxy" Oct 30 15:58:27.852199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851542 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d62b438-6f71-4ea0-9493-3ce8f382fe21" containerName="istio-proxy" Oct 30 15:58:27.852199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851550 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f366feb-99b7-4023-bc5d-72a66f8eaefe" containerName="istio-proxy" Oct 30 15:58:27.852199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851562 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c0f1fd-4e1c-4f99-a59d-8e5d711a1a84" containerName="istio-proxy" Oct 30 15:58:27.852199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851569 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e64b4190-cdf8-45e6-83b5-aa6081a311f0" containerName="istio-proxy" Oct 30 15:58:27.852199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851576 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="0728c928-7b7d-4ed6-a3df-5e35727c71b0" containerName="istio-proxy" Oct 30 15:58:27.852199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.851581 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c7ea812-ccc6-44d1-88a9-425edea737e5" containerName="payload-pre-processing" Oct 30 15:58:27.856963 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.856929 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.878377 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.878323 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.878566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.878408 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.878566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.878493 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.878566 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.878541 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.878802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.878772 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.879003 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.878966 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.879122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.879050 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.879122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.879084 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnqwd\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-kube-api-access-tnqwd\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.879122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.879117 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.887554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.887514 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw"] Oct 30 15:58:27.980181 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980144 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980376 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980192 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980376 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980242 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980376 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980266 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980376 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980330 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980379 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980428 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980457 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnqwd\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-kube-api-access-tnqwd\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980483 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980750 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980613 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-certs\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980750 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980643 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980750 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980724 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-data\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.980937 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.980794 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-credential-socket\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.981272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.981245 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istiod-ca-cert\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.982977 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.982949 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-envoy\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:27.983121 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:27.983100 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-podinfo\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:28.009362 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.009331 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-token\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:28.009636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.009607 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnqwd\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-kube-api-access-tnqwd\") pod \"e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:28.020055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.020015 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg"] Oct 30 15:58:28.026685 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.026657 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.044812 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.044767 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg"] Oct 30 15:58:28.045722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.045678 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:58:28.046513 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.046488 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:58:28.081638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.081593 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b97cq\" (UniqueName: \"kubernetes.io/projected/6fc868aa-311e-400f-bf44-808911468167-kube-api-access-b97cq\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.081901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.081740 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc868aa-311e-400f-bf44-808911468167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.082078 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.082054 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc868aa-311e-400f-bf44-808911468167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.128621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.128529 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:58:28.128621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.128556 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:58:28.183256 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.183206 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc868aa-311e-400f-bf44-808911468167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.183459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.183324 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b97cq\" (UniqueName: \"kubernetes.io/projected/6fc868aa-311e-400f-bf44-808911468167-kube-api-access-b97cq\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.183459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.183349 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:28.183459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.183436 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc868aa-311e-400f-bf44-808911468167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.184056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.184034 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc868aa-311e-400f-bf44-808911468167-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.185852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.185813 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc868aa-311e-400f-bf44-808911468167-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.208622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.208594 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b97cq\" (UniqueName: \"kubernetes.io/projected/6fc868aa-311e-400f-bf44-808911468167-kube-api-access-b97cq\") pod \"maas-api-revoke-sub-a1fd93a5-wmfhg\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.359047 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.359006 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:28.531257 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:28.531213 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdba6667_d2cb_41d3_bafb_e0d04c8fc808.slice/crio-112016c8f9a1f522eecf196a6cd4bcfd7ec5c0e462bfef424bf36cd413304cca WatchSource:0}: Error finding container 112016c8f9a1f522eecf196a6cd4bcfd7ec5c0e462bfef424bf36cd413304cca: Status 404 returned error can't find the container with id 112016c8f9a1f522eecf196a6cd4bcfd7ec5c0e462bfef424bf36cd413304cca Oct 30 15:58:28.533984 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.533956 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw"] Oct 30 15:58:28.534228 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.534194 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:28.534281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.534267 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:28.534326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.534313 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:28.677473 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:28.677441 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc868aa_311e_400f_bf44_808911468167.slice/crio-fa544bcaf85190d933f21f2aa6c6fc392dba9e6163a0580121cf0575d3f3bc13 WatchSource:0}: Error finding container fa544bcaf85190d933f21f2aa6c6fc392dba9e6163a0580121cf0575d3f3bc13: Status 404 returned error can't find the container with id fa544bcaf85190d933f21f2aa6c6fc392dba9e6163a0580121cf0575d3f3bc13 Oct 30 15:58:28.678750 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:28.678566 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg"] Oct 30 15:58:29.185640 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:29.185595 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" event={"ID":"fdba6667-d2cb-41d3-bafb-e0d04c8fc808","Type":"ContainerStarted","Data":"10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248"} Oct 30 15:58:29.185868 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:29.185665 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" event={"ID":"fdba6667-d2cb-41d3-bafb-e0d04c8fc808","Type":"ContainerStarted","Data":"112016c8f9a1f522eecf196a6cd4bcfd7ec5c0e462bfef424bf36cd413304cca"} Oct 30 15:58:29.188005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:29.187973 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" event={"ID":"6fc868aa-311e-400f-bf44-808911468167","Type":"ContainerStarted","Data":"29e11605a83b6839fc133c556f895ed6ee025a174f1eb25cbf6ff2c8f70f0bc5"} Oct 30 15:58:29.188165 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:29.188021 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" event={"ID":"6fc868aa-311e-400f-bf44-808911468167","Type":"ContainerStarted","Data":"fa544bcaf85190d933f21f2aa6c6fc392dba9e6163a0580121cf0575d3f3bc13"} Oct 30 15:58:29.236016 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:29.235899 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podStartSLOduration=2.235883219 podStartE2EDuration="2.235883219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:29.234375517 +0000 UTC m=+1217.933648676" watchObservedRunningTime="2026-10-30 15:58:29.235883219 +0000 UTC m=+1217.935156375" Oct 30 15:58:29.294793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:29.294729 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" podStartSLOduration=2.294713797 podStartE2EDuration="2.294713797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:29.293044919 +0000 UTC m=+1217.992318088" watchObservedRunningTime="2026-10-30 15:58:29.294713797 +0000 UTC m=+1217.993986963" Oct 30 15:58:30.183775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:30.183733 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:30.185648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:30.185614 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 15:58:30.185806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:30.185682 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 15:58:30.200857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:30.200808 4401 generic.go:356] "Generic (PLEG): container finished" podID="6fc868aa-311e-400f-bf44-808911468167" containerID="29e11605a83b6839fc133c556f895ed6ee025a174f1eb25cbf6ff2c8f70f0bc5" exitCode=0 Oct 30 15:58:30.200857 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:30.200851 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" event={"ID":"6fc868aa-311e-400f-bf44-808911468167","Type":"ContainerDied","Data":"29e11605a83b6839fc133c556f895ed6ee025a174f1eb25cbf6ff2c8f70f0bc5"} Oct 30 15:58:31.185130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.184211 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 15:58:31.185130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.184276 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 15:58:31.427344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.427311 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:31.507159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.507071 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs"] Oct 30 15:58:31.507734 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.507713 4401 cpu_manager.go:414] "containerMap: removing container" podUID="6fc868aa-311e-400f-bf44-808911468167" containerName="revoke-keys" Oct 30 15:58:31.507734 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.507733 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc868aa-311e-400f-bf44-808911468167" containerName="revoke-keys" Oct 30 15:58:31.507929 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.507872 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc868aa-311e-400f-bf44-808911468167" containerName="revoke-keys" Oct 30 15:58:31.511927 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.511861 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.538472 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.538428 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fc868aa-311e-400f-bf44-808911468167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc868aa-311e-400f-bf44-808911468167-maas-api-openshift-ca\") pod \"6fc868aa-311e-400f-bf44-808911468167\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " Oct 30 15:58:31.538926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.538887 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fc868aa-311e-400f-bf44-808911468167-maas-api-openshift-ca" pod "6fc868aa-311e-400f-bf44-808911468167" (UID: "6fc868aa-311e-400f-bf44-808911468167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:31.539027 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.538948 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc868aa-311e-400f-bf44-808911468167-kube-api-access-b97cq\" (UniqueName: \"kubernetes.io/projected/6fc868aa-311e-400f-bf44-808911468167-kube-api-access-b97cq\") pod \"6fc868aa-311e-400f-bf44-808911468167\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " Oct 30 15:58:31.539094 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.539023 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fc868aa-311e-400f-bf44-808911468167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc868aa-311e-400f-bf44-808911468167-maas-api-xks-ca\") pod \"6fc868aa-311e-400f-bf44-808911468167\" (UID: \"6fc868aa-311e-400f-bf44-808911468167\") " Oct 30 15:58:31.539765 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.539725 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.539938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.539896 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.539996 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.539958 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdtn\" (UniqueName: \"kubernetes.io/projected/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-kube-api-access-xtdtn\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.541042 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.540077 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fc868aa-311e-400f-bf44-808911468167-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:31.577801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.575491 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc868aa-311e-400f-bf44-808911468167-kube-api-access-b97cq" pod "6fc868aa-311e-400f-bf44-808911468167" (UID: "6fc868aa-311e-400f-bf44-808911468167"). InnerVolumeSpecName "kube-api-access-b97cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:31.591899 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.581705 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fc868aa-311e-400f-bf44-808911468167-maas-api-xks-ca" pod "6fc868aa-311e-400f-bf44-808911468167" (UID: "6fc868aa-311e-400f-bf44-808911468167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:31.591899 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.585609 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs"] Oct 30 15:58:31.641163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.641122 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.641344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.641212 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.641344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.641246 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdtn\" (UniqueName: \"kubernetes.io/projected/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-kube-api-access-xtdtn\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.641414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.641358 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b97cq\" (UniqueName: \"kubernetes.io/projected/6fc868aa-311e-400f-bf44-808911468167-kube-api-access-b97cq\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:31.641414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.641370 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fc868aa-311e-400f-bf44-808911468167-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:31.642179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.642151 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.643748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.643714 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.679535 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.679503 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdtn\" (UniqueName: \"kubernetes.io/projected/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-kube-api-access-xtdtn\") pod \"maas-api-revoke-sub-1fd258fb-zr5xs\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:31.855124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:31.855073 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:32.032942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.032897 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw"] Oct 30 15:58:32.038297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.038258 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-539d3120-gscxw"] Oct 30 15:58:32.184134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.184042 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 15:58:32.184134 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.184121 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 15:58:32.194547 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:32.194514 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b71dd18_7858_44b4_ad75_9d57e2f8c90e.slice/crio-4a7aa84e5eaa90f642025e1b3c86059b367126dcba4f819d0330dfcda94877ae WatchSource:0}: Error finding container 4a7aa84e5eaa90f642025e1b3c86059b367126dcba4f819d0330dfcda94877ae: Status 404 returned error can't find the container with id 4a7aa84e5eaa90f642025e1b3c86059b367126dcba4f819d0330dfcda94877ae Oct 30 15:58:32.195738 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.195710 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs"] Oct 30 15:58:32.240200 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.240171 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" Oct 30 15:58:32.240200 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.240188 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg" event={"ID":"6fc868aa-311e-400f-bf44-808911468167","Type":"ContainerDied","Data":"fa544bcaf85190d933f21f2aa6c6fc392dba9e6163a0580121cf0575d3f3bc13"} Oct 30 15:58:32.240420 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.240240 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa544bcaf85190d933f21f2aa6c6fc392dba9e6163a0580121cf0575d3f3bc13" Oct 30 15:58:32.241866 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.241817 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" event={"ID":"7b71dd18-7858-44b4-ad75-9d57e2f8c90e","Type":"ContainerStarted","Data":"4a7aa84e5eaa90f642025e1b3c86059b367126dcba4f819d0330dfcda94877ae"} Oct 30 15:58:32.624459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.624431 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:58:32.679512 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679460 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-socket\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.679700 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679540 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d05b27-3f87-4a35-8502-3ddedaa5456e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d05b27-3f87-4a35-8502-3ddedaa5456e-istiod-ca-cert\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.679700 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679666 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-podinfo\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.679819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679702 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-token\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-token\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.679819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679761 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-kube-api-access-bgnwc\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-kube-api-access-bgnwc\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.679956 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679858 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-certs\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.679956 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679889 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-credential-socket\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.679956 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.679917 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-envoy\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.680098 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.680054 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-data\") pod \"25d05b27-3f87-4a35-8502-3ddedaa5456e\" (UID: \"25d05b27-3f87-4a35-8502-3ddedaa5456e\") " Oct 30 15:58:32.680942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.680568 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d05b27-3f87-4a35-8502-3ddedaa5456e-istiod-ca-cert" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:32.680942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.680848 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25d05b27-3f87-4a35-8502-3ddedaa5456e-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.680942 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.680906 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-socket" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:32.681366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.680968 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-certs" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:32.681366 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.681044 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-credential-socket" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:32.681512 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.681378 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-data" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:32.685807 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.685760 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-podinfo" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:32.686014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.685888 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-token" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:32.687122 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.686906 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-envoy" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:32.688655 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.688612 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-kube-api-access-bgnwc" pod "25d05b27-3f87-4a35-8502-3ddedaa5456e" (UID: "25d05b27-3f87-4a35-8502-3ddedaa5456e"). InnerVolumeSpecName "kube-api-access-bgnwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:32.781873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781757 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.781873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781794 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.781873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781809 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.781873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781864 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnwc\" (UniqueName: \"kubernetes.io/projected/25d05b27-3f87-4a35-8502-3ddedaa5456e-kube-api-access-bgnwc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.782137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781881 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.782137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781893 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.782137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781904 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:32.782137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:32.781916 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25d05b27-3f87-4a35-8502-3ddedaa5456e-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:33.155422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.155387 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh"] Oct 30 15:58:33.156775 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.156743 4401 cpu_manager.go:414] "containerMap: removing container" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" Oct 30 15:58:33.156995 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.156979 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" Oct 30 15:58:33.157127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.157117 4401 cpu_manager.go:414] "containerMap: removing container" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" Oct 30 15:58:33.157223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.157201 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" Oct 30 15:58:33.157531 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.157514 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" Oct 30 15:58:33.173701 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.157645 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerName="istio-proxy" Oct 30 15:58:33.190850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.185504 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 15:58:33.190850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.185575 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 15:58:33.197217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.197180 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.214202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.214148 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh"] Oct 30 15:58:33.286994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.286959 4401 generic.go:356] "Generic (PLEG): container finished" podID="25d05b27-3f87-4a35-8502-3ddedaa5456e" containerID="07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802" exitCode=0 Oct 30 15:58:33.287124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.287010 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" Oct 30 15:58:33.287124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.287041 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" event={"ID":"25d05b27-3f87-4a35-8502-3ddedaa5456e","Type":"ContainerDied","Data":"07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802"} Oct 30 15:58:33.287124 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.287106 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2" event={"ID":"25d05b27-3f87-4a35-8502-3ddedaa5456e","Type":"ContainerDied","Data":"7b3b177b4e4b1b9389e79d5b8c1b7b7c86d0d13b088244ee114a828daf4d7137"} Oct 30 15:58:33.287266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.287135 4401 scope.go:122] "RemoveContainer" containerID="07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802" Oct 30 15:58:33.289293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.289234 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" event={"ID":"7b71dd18-7858-44b4-ad75-9d57e2f8c90e","Type":"ContainerStarted","Data":"d56babca00563a2bb965b8a1bcef7b9fc2be779d1edfacf0645437868e33247a"} Oct 30 15:58:33.292672 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.292648 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.292771 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.292713 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcqdr\" (UniqueName: \"kubernetes.io/projected/7cf706a2-8886-4104-bd12-b13608ae8cbd-kube-api-access-wcqdr\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.292811 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.292797 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.340711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.340687 4401 scope.go:122] "RemoveContainer" containerID="6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5" Oct 30 15:58:33.382514 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.382472 4401 scope.go:122] "RemoveContainer" containerID="07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802" Oct 30 15:58:33.382890 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:33.382866 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802\": container with ID starting with 07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802 not found: ID does not exist" containerID="07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802" Oct 30 15:58:33.382960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.382900 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802"} err="failed to get container status \"07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802\": rpc error: code = NotFound desc = could not find container \"07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802\": container with ID starting with 07b7ad90763786b53563411e338cfe5324c790587c3421fece10c95234d47802 not found: ID does not exist" Oct 30 15:58:33.382960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.382919 4401 scope.go:122] "RemoveContainer" containerID="6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5" Oct 30 15:58:33.383230 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:33.383209 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5\": container with ID starting with 6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5 not found: ID does not exist" containerID="6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5" Oct 30 15:58:33.383294 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.383239 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5"} err="failed to get container status \"6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5\": rpc error: code = NotFound desc = could not find container \"6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5\": container with ID starting with 6be7896aef7bf27bdbd9958544478700d64ee122b029c577fb0e15f28f18e3d5 not found: ID does not exist" Oct 30 15:58:33.394581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.394536 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcqdr\" (UniqueName: \"kubernetes.io/projected/7cf706a2-8886-4104-bd12-b13608ae8cbd-kube-api-access-wcqdr\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.394758 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.394587 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.394862 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.394811 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.395739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.395708 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.397071 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.397051 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.415668 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.415579 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2"] Oct 30 15:58:33.424202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.424168 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcqdr\" (UniqueName: \"kubernetes.io/projected/7cf706a2-8886-4104-bd12-b13608ae8cbd-kube-api-access-wcqdr\") pod \"maas-api-revoke-sub-48a7e5a1-g5srh\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.429335 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.429306 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b81e87-50cb-416b-8670-59c6329084e1" path="/var/lib/kubelet/pods/d8b81e87-50cb-416b-8670-59c6329084e1/volumes" Oct 30 15:58:33.429670 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.429655 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-brcq2"] Oct 30 15:58:33.529780 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.529745 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:33.878998 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:33.878962 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cf706a2_8886_4104_bd12_b13608ae8cbd.slice/crio-9ec05ff6bc9cfbe1a04b1157f44d814bce8b2b7a2b9c06054eac5571080dc737 WatchSource:0}: Error finding container 9ec05ff6bc9cfbe1a04b1157f44d814bce8b2b7a2b9c06054eac5571080dc737: Status 404 returned error can't find the container with id 9ec05ff6bc9cfbe1a04b1157f44d814bce8b2b7a2b9c06054eac5571080dc737 Oct 30 15:58:33.882015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:33.881984 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh"] Oct 30 15:58:34.184414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.184325 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 15:58:34.184414 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.184378 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 15:58:34.302787 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.302738 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" event={"ID":"7cf706a2-8886-4104-bd12-b13608ae8cbd","Type":"ContainerStarted","Data":"9ec05ff6bc9cfbe1a04b1157f44d814bce8b2b7a2b9c06054eac5571080dc737"} Oct 30 15:58:34.304625 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.304580 4401 generic.go:356] "Generic (PLEG): container finished" podID="7b71dd18-7858-44b4-ad75-9d57e2f8c90e" containerID="d56babca00563a2bb965b8a1bcef7b9fc2be779d1edfacf0645437868e33247a" exitCode=0 Oct 30 15:58:34.304798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.304765 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" event={"ID":"7b71dd18-7858-44b4-ad75-9d57e2f8c90e","Type":"ContainerDied","Data":"d56babca00563a2bb965b8a1bcef7b9fc2be779d1edfacf0645437868e33247a"} Oct 30 15:58:34.518961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.518930 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:34.611934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.611893 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-xks-ca\") pod \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " Oct 30 15:58:34.612116 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.612035 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-kube-api-access-xtdtn\" (UniqueName: \"kubernetes.io/projected/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-kube-api-access-xtdtn\") pod \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " Oct 30 15:58:34.612211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.612199 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-openshift-ca\") pod \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\" (UID: \"7b71dd18-7858-44b4-ad75-9d57e2f8c90e\") " Oct 30 15:58:34.612563 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.612521 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-openshift-ca" pod "7b71dd18-7858-44b4-ad75-9d57e2f8c90e" (UID: "7b71dd18-7858-44b4-ad75-9d57e2f8c90e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:34.614149 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.614116 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-xks-ca" pod "7b71dd18-7858-44b4-ad75-9d57e2f8c90e" (UID: "7b71dd18-7858-44b4-ad75-9d57e2f8c90e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:34.614319 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.614275 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-kube-api-access-xtdtn" pod "7b71dd18-7858-44b4-ad75-9d57e2f8c90e" (UID: "7b71dd18-7858-44b4-ad75-9d57e2f8c90e"). InnerVolumeSpecName "kube-api-access-xtdtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:34.713484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.713442 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:34.713484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.713479 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:34.713484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:34.713490 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtdtn\" (UniqueName: \"kubernetes.io/projected/7b71dd18-7858-44b4-ad75-9d57e2f8c90e-kube-api-access-xtdtn\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:35.185467 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.185427 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 15:58:35.185702 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.185485 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 15:58:35.318774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.318736 4401 generic.go:356] "Generic (PLEG): container finished" podID="7cf706a2-8886-4104-bd12-b13608ae8cbd" containerID="42609da5c396872a3b78715e05bf2fffe7e1e6b3d1d8dafe53e4835e6b3081c2" exitCode=0 Oct 30 15:58:35.319227 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.318845 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" event={"ID":"7cf706a2-8886-4104-bd12-b13608ae8cbd","Type":"ContainerDied","Data":"42609da5c396872a3b78715e05bf2fffe7e1e6b3d1d8dafe53e4835e6b3081c2"} Oct 30 15:58:35.320621 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.320598 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" Oct 30 15:58:35.320739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.320624 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs" event={"ID":"7b71dd18-7858-44b4-ad75-9d57e2f8c90e","Type":"ContainerDied","Data":"4a7aa84e5eaa90f642025e1b3c86059b367126dcba4f819d0330dfcda94877ae"} Oct 30 15:58:35.320739 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.320674 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a7aa84e5eaa90f642025e1b3c86059b367126dcba4f819d0330dfcda94877ae" Oct 30 15:58:35.434449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.434404 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d05b27-3f87-4a35-8502-3ddedaa5456e" path="/var/lib/kubelet/pods/25d05b27-3f87-4a35-8502-3ddedaa5456e/volumes" Oct 30 15:58:35.485953 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.485916 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf"] Oct 30 15:58:35.486174 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.486134 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" containerID="cri-o://a90b212cfcc710195550c779b29f9355ff6ff714fa8538fc6c7f3dcbdf86b014" gracePeriod=30 Oct 30 15:58:35.493759 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.493722 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj"] Oct 30 15:58:35.494109 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.494036 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" containerID="cri-o://42b4b940b771e9d1570abcb67747d9fa63092005126a250c2237758d7209dd73" gracePeriod=30 Oct 30 15:58:35.510703 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.510656 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28"] Oct 30 15:58:35.510923 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.510901 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" containerID="cri-o://2093ecfc260ab011160bfae9b4642fe2788969f47c7319977f85ba0ee2d8d5bd" gracePeriod=30 Oct 30 15:58:35.519818 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.519774 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw"] Oct 30 15:58:35.520085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.520057 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" containerID="cri-o://c8780163b9f8e75f85b8e2992366edcb935fd7e42a21e1d2778c0d3ce135f092" gracePeriod=30 Oct 30 15:58:35.636882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.635164 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:58:35.636882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.635231 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:58:35.841524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.841436 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:58:35.841524 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:35.841502 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:58:36.332460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.332423 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:36.332951 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.332934 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:36.333785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.333760 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:58:36.515962 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.515930 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:36.552073 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.552034 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cf706a2-8886-4104-bd12-b13608ae8cbd-kube-api-access-wcqdr\" (UniqueName: \"kubernetes.io/projected/7cf706a2-8886-4104-bd12-b13608ae8cbd-kube-api-access-wcqdr\") pod \"7cf706a2-8886-4104-bd12-b13608ae8cbd\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " Oct 30 15:58:36.552374 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.552154 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-xks-ca\") pod \"7cf706a2-8886-4104-bd12-b13608ae8cbd\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " Oct 30 15:58:36.552374 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.552252 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-openshift-ca\") pod \"7cf706a2-8886-4104-bd12-b13608ae8cbd\" (UID: \"7cf706a2-8886-4104-bd12-b13608ae8cbd\") " Oct 30 15:58:36.552895 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.552814 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-openshift-ca" pod "7cf706a2-8886-4104-bd12-b13608ae8cbd" (UID: "7cf706a2-8886-4104-bd12-b13608ae8cbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:36.554628 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.554591 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-xks-ca" pod "7cf706a2-8886-4104-bd12-b13608ae8cbd" (UID: "7cf706a2-8886-4104-bd12-b13608ae8cbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:36.554746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.554669 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf706a2-8886-4104-bd12-b13608ae8cbd-kube-api-access-wcqdr" pod "7cf706a2-8886-4104-bd12-b13608ae8cbd" (UID: "7cf706a2-8886-4104-bd12-b13608ae8cbd"). InnerVolumeSpecName "kube-api-access-wcqdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:36.654398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.654360 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcqdr\" (UniqueName: \"kubernetes.io/projected/7cf706a2-8886-4104-bd12-b13608ae8cbd-kube-api-access-wcqdr\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:36.654398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.654393 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:36.654398 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:36.654403 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cf706a2-8886-4104-bd12-b13608ae8cbd-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:37.344199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:37.344161 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" Oct 30 15:58:37.344199 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:37.344187 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh" event={"ID":"7cf706a2-8886-4104-bd12-b13608ae8cbd","Type":"ContainerDied","Data":"9ec05ff6bc9cfbe1a04b1157f44d814bce8b2b7a2b9c06054eac5571080dc737"} Oct 30 15:58:37.344629 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:37.344248 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ec05ff6bc9cfbe1a04b1157f44d814bce8b2b7a2b9c06054eac5571080dc737" Oct 30 15:58:38.114306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:38.114257 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:58:39.080902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:39.079730 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg"] Oct 30 15:58:39.146849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:39.145839 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31fe52ec-lnmvg"] Oct 30 15:58:39.667853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:39.638933 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b649fdd2-99c9-4a74-9da8-c4d872965e1c" path="/var/lib/kubelet/pods/b649fdd2-99c9-4a74-9da8-c4d872965e1c/volumes" Oct 30 15:58:40.161040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:40.160995 4401 patch_prober.go:29] interesting pod/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:58:40.161636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:40.161599 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:58:40.726853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:40.723674 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:58:41.067229 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.066044 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" podUID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:58:41.176318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.174949 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 15:58:41.177432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.176966 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:58:41.178488 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.178091 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:58:41.273853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.270962 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:58:41.521955 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.513665 4401 generic.go:356] "Generic (PLEG): container finished" podID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerID="c8780163b9f8e75f85b8e2992366edcb935fd7e42a21e1d2778c0d3ce135f092" exitCode=0 Oct 30 15:58:41.521955 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.513800 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" event={"ID":"cf8176a4-c357-45be-b324-ccbb8c53ee36","Type":"ContainerDied","Data":"c8780163b9f8e75f85b8e2992366edcb935fd7e42a21e1d2778c0d3ce135f092"} Oct 30 15:58:41.553576 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.553526 4401 generic.go:356] "Generic (PLEG): container finished" podID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerID="42b4b940b771e9d1570abcb67747d9fa63092005126a250c2237758d7209dd73" exitCode=0 Oct 30 15:58:41.554080 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.554039 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" event={"ID":"65b4e65e-4696-4b8d-a5a9-f6ab671149e8","Type":"ContainerDied","Data":"42b4b940b771e9d1570abcb67747d9fa63092005126a250c2237758d7209dd73"} Oct 30 15:58:41.568870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.560090 4401 generic.go:356] "Generic (PLEG): container finished" podID="7e3faa20-1ade-4a35-812b-997c25755b10" containerID="2093ecfc260ab011160bfae9b4642fe2788969f47c7319977f85ba0ee2d8d5bd" exitCode=0 Oct 30 15:58:41.568870 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.560253 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" event={"ID":"7e3faa20-1ade-4a35-812b-997c25755b10","Type":"ContainerDied","Data":"2093ecfc260ab011160bfae9b4642fe2788969f47c7319977f85ba0ee2d8d5bd"} Oct 30 15:58:41.583860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.576600 4401 generic.go:356] "Generic (PLEG): container finished" podID="35eca30d-cb5e-417b-ae35-6faefa966211" containerID="a90b212cfcc710195550c779b29f9355ff6ff714fa8538fc6c7f3dcbdf86b014" exitCode=0 Oct 30 15:58:41.583860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.576661 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" event={"ID":"35eca30d-cb5e-417b-ae35-6faefa966211","Type":"ContainerDied","Data":"a90b212cfcc710195550c779b29f9355ff6ff714fa8538fc6c7f3dcbdf86b014"} Oct 30 15:58:41.807040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.805921 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded" Oct 30 15:58:41.807040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.806884 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:58:41.925908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:41.921248 4401 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Oct 30 15:58:42.091718 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.086929 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" podUID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.143:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:58:42.091718 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.089624 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:42.211627 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.211567 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 15:58:42.224761 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.224708 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": context deadline exceeded" Oct 30 15:58:42.253468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.247980 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:58:42.253468 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.248845 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:58:42.314863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.314778 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": context deadline exceeded" Oct 30 15:58:42.417066 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.315923 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:58:42.584315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.579901 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb"] Oct 30 15:58:42.584315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.580990 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7b71dd18-7858-44b4-ad75-9d57e2f8c90e" containerName="revoke-keys" Oct 30 15:58:42.584315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.581016 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b71dd18-7858-44b4-ad75-9d57e2f8c90e" containerName="revoke-keys" Oct 30 15:58:42.584315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.581040 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7cf706a2-8886-4104-bd12-b13608ae8cbd" containerName="revoke-keys" Oct 30 15:58:42.584315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.581048 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cf706a2-8886-4104-bd12-b13608ae8cbd" containerName="revoke-keys" Oct 30 15:58:42.584315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.581194 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b71dd18-7858-44b4-ad75-9d57e2f8c90e" containerName="revoke-keys" Oct 30 15:58:42.584315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.581207 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cf706a2-8886-4104-bd12-b13608ae8cbd" containerName="revoke-keys" Oct 30 15:58:42.586371 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.586342 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.617905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.617870 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:58:42.618189 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.618168 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:58:42.618433 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.618414 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:58:42.632129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.621064 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:58:42.632129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.630701 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:58:42.633115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.633079 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb"] Oct 30 15:58:42.633635 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.633613 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:58:42.858107 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.826301 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": context deadline exceeded" Oct 30 15:58:42.874089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.858750 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:58:42.874089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.861447 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:58:42.874089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.865881 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.874089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.865959 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh7c7\" (UniqueName: \"kubernetes.io/projected/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-kube-api-access-gh7c7\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.874089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.866069 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.911969 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.911925 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:58:42.972314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.972278 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.972805 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.972778 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.973349 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.973326 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh7c7\" (UniqueName: \"kubernetes.io/projected/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-kube-api-access-gh7c7\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.973894 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.973188 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.979484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.979451 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:42.992495 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.992439 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:58:42.996271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.996066 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:58:42.999022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.998998 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:58:43.000020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:42.999993 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh7c7\" (UniqueName: \"kubernetes.io/projected/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-kube-api-access-gh7c7\") pod \"maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:43.002336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.002311 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:58:43.176425 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176386 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-certs\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.176713 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176686 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-kube-api-access-z5jkt\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-kube-api-access-z5jkt\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.176900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176718 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-certs" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.176900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176763 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-data\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.176900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176785 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-certs\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.176900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176815 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-envoy\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.176900 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176878 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-credential-socket\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176911 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-credential-socket\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176946 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-envoy\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.176975 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e3faa20-1ade-4a35-812b-997c25755b10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3faa20-1ade-4a35-812b-997c25755b10-istio-podinfo\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177024 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-data\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177049 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-certs\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177083 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-socket\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177115 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-podinfo\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177139 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-socket\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177198 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-kube-api-access-p6rv2\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-kube-api-access-p6rv2\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.177221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177235 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-socket\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177268 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35eca30d-cb5e-417b-ae35-6faefa966211-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eca30d-cb5e-417b-ae35-6faefa966211-istio-podinfo\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177310 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-token\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-token\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177336 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-credential-socket\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177381 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-envoy\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177412 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-istio-token\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-istio-token\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177452 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-podinfo\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177487 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-data\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.177997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.177563 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-envoy\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.178897 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.178797 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-data" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.183129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.182593 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-data" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.183129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183000 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-token\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-token\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.183129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183061 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-kube-api-access-n2d8d\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-kube-api-access-n2d8d\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.183129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183111 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-socket\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183146 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-certs\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183198 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-kube-api-access-g8xq2\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-kube-api-access-g8xq2\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183225 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istiod-ca-cert\") pod \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\" (UID: \"65b4e65e-4696-4b8d-a5a9-f6ab671149e8\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183253 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e3faa20-1ade-4a35-812b-997c25755b10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3faa20-1ade-4a35-812b-997c25755b10-istiod-ca-cert\") pod \"7e3faa20-1ade-4a35-812b-997c25755b10\" (UID: \"7e3faa20-1ade-4a35-812b-997c25755b10\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183330 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eca30d-cb5e-417b-ae35-6faefa966211-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eca30d-cb5e-417b-ae35-6faefa966211-istiod-ca-cert\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183422 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-data\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183474 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-credential-socket\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.183536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183505 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf8176a4-c357-45be-b324-ccbb8c53ee36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8176a4-c357-45be-b324-ccbb8c53ee36-istiod-ca-cert\") pod \"cf8176a4-c357-45be-b324-ccbb8c53ee36\" (UID: \"cf8176a4-c357-45be-b324-ccbb8c53ee36\") " Oct 30 15:58:43.184099 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183563 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-istio-token\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-istio-token\") pod \"35eca30d-cb5e-417b-ae35-6faefa966211\" (UID: \"35eca30d-cb5e-417b-ae35-6faefa966211\") " Oct 30 15:58:43.186183 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.186153 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.186183 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.186187 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.186567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.186201 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183218 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-certs" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.183899 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-socket" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.184100 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-socket" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.184945 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-kube-api-access-z5jkt" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "kube-api-access-z5jkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.185131 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-credential-socket" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.185228 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-socket" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.185338 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-envoy" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.185681 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-socket" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.185878 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-credential-socket" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.186632 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-credential-socket" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.186796 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-certs" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.187806 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-certs" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.188291 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-podinfo" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:43.188485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.188294 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eca30d-cb5e-417b-ae35-6faefa966211-istiod-ca-cert" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:43.189341 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.188715 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istiod-ca-cert" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:43.189550 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.189435 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-data" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.189620 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.189593 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-token" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.189863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.189794 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e3faa20-1ade-4a35-812b-997c25755b10-istiod-ca-cert" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:43.191241 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.191209 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-kube-api-access-p6rv2" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "kube-api-access-p6rv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.192046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.191754 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-token" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.192149 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.192106 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-credential-socket" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.192215 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.192161 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-envoy" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.192384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.192340 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-data" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.192527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.192502 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf8176a4-c357-45be-b324-ccbb8c53ee36-istiod-ca-cert" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:43.193554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.193528 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-envoy" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.194972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.194943 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-istio-token" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.195263 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.195235 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-kube-api-access-n2d8d" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "kube-api-access-n2d8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.198340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.198229 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-envoy" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:58:43.198340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.198251 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-istio-token" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.198340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.198255 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-podinfo" pod "cf8176a4-c357-45be-b324-ccbb8c53ee36" (UID: "cf8176a4-c357-45be-b324-ccbb8c53ee36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:43.198340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.198296 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-kube-api-access-g8xq2" pod "65b4e65e-4696-4b8d-a5a9-f6ab671149e8" (UID: "65b4e65e-4696-4b8d-a5a9-f6ab671149e8"). InnerVolumeSpecName "kube-api-access-g8xq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:43.198340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.198320 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e3faa20-1ade-4a35-812b-997c25755b10-istio-podinfo" pod "7e3faa20-1ade-4a35-812b-997c25755b10" (UID: "7e3faa20-1ade-4a35-812b-997c25755b10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:43.199896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.199867 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35eca30d-cb5e-417b-ae35-6faefa966211-istio-podinfo" pod "35eca30d-cb5e-417b-ae35-6faefa966211" (UID: "35eca30d-cb5e-417b-ae35-6faefa966211"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:58:43.233881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.233840 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288152 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35eca30d-cb5e-417b-ae35-6faefa966211-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288181 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288190 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288202 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf8176a4-c357-45be-b324-ccbb8c53ee36-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288212 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288220 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5jkt\" (UniqueName: \"kubernetes.io/projected/35eca30d-cb5e-417b-ae35-6faefa966211-kube-api-access-z5jkt\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288228 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288237 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288245 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288253 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288261 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288269 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e3faa20-1ade-4a35-812b-997c25755b10-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288276 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288284 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288292 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288300 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288308 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6rv2\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-kube-api-access-p6rv2\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288317 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288328 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35eca30d-cb5e-417b-ae35-6faefa966211-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288336 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288345 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf8176a4-c357-45be-b324-ccbb8c53ee36-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288354 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288362 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e3faa20-1ade-4a35-812b-997c25755b10-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288370 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf8176a4-c357-45be-b324-ccbb8c53ee36-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288378 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288386 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35eca30d-cb5e-417b-ae35-6faefa966211-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288393 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288400 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2d8d\" (UniqueName: \"kubernetes.io/projected/cf8176a4-c357-45be-b324-ccbb8c53ee36-kube-api-access-n2d8d\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288409 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288417 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e3faa20-1ade-4a35-812b-997c25755b10-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288426 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xq2\" (UniqueName: \"kubernetes.io/projected/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-kube-api-access-g8xq2\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288435 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65b4e65e-4696-4b8d-a5a9-f6ab671149e8-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.288790 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.288443 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e3faa20-1ade-4a35-812b-997c25755b10-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:43.553941 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:43.553907 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc30ecf6_f9d2_45af_aaaf_a5d63e52fd37.slice/crio-49ab0ab79c9207a36c11e4f3b633dc694a7fd61164345a1af0424cde16386ede WatchSource:0}: Error finding container 49ab0ab79c9207a36c11e4f3b633dc694a7fd61164345a1af0424cde16386ede: Status 404 returned error can't find the container with id 49ab0ab79c9207a36c11e4f3b633dc694a7fd61164345a1af0424cde16386ede Oct 30 15:58:43.555403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.555372 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb"] Oct 30 15:58:43.926446 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.926413 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" Oct 30 15:58:43.926645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.926412 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw" event={"ID":"cf8176a4-c357-45be-b324-ccbb8c53ee36","Type":"ContainerDied","Data":"549e61829518b61242df4c7c495280c95202a0bb5b7b3dd710ec2d445d2ec6fe"} Oct 30 15:58:43.926645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.926559 4401 scope.go:122] "RemoveContainer" containerID="c8780163b9f8e75f85b8e2992366edcb935fd7e42a21e1d2778c0d3ce135f092" Oct 30 15:58:43.928198 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.928176 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" Oct 30 15:58:43.928310 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.928195 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj" event={"ID":"65b4e65e-4696-4b8d-a5a9-f6ab671149e8","Type":"ContainerDied","Data":"99cd352f9f690c35ab6bcbbeac911d6699a2dc72a867c3b25b185d7cd426a438"} Oct 30 15:58:43.930076 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.930049 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" Oct 30 15:58:43.930198 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.930090 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28" event={"ID":"7e3faa20-1ade-4a35-812b-997c25755b10","Type":"ContainerDied","Data":"f795444981348f60a0788721a97602512f80e0d706b9b3893f1873f540b2d055"} Oct 30 15:58:43.931981 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.931959 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" event={"ID":"35eca30d-cb5e-417b-ae35-6faefa966211","Type":"ContainerDied","Data":"fc23f8a84f3cf2695d8056b49c70e74c8f6ce6e1b8cc2ef4a31ebe42fa0d3143"} Oct 30 15:58:43.932100 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.931969 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf" Oct 30 15:58:43.933488 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.933465 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" event={"ID":"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37","Type":"ContainerStarted","Data":"49ab0ab79c9207a36c11e4f3b633dc694a7fd61164345a1af0424cde16386ede"} Oct 30 15:58:43.940282 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.940262 4401 scope.go:122] "RemoveContainer" containerID="42b4b940b771e9d1570abcb67747d9fa63092005126a250c2237758d7209dd73" Oct 30 15:58:43.951936 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.951914 4401 scope.go:122] "RemoveContainer" containerID="2093ecfc260ab011160bfae9b4642fe2788969f47c7319977f85ba0ee2d8d5bd" Oct 30 15:58:43.963691 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:43.963668 4401 scope.go:122] "RemoveContainer" containerID="a90b212cfcc710195550c779b29f9355ff6ff714fa8538fc6c7f3dcbdf86b014" Oct 30 15:58:44.013618 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.013579 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf"] Oct 30 15:58:44.032710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.032677 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-kz7pf"] Oct 30 15:58:44.083710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.083621 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28"] Oct 30 15:58:44.089251 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.089224 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-zrt28"] Oct 30 15:58:44.132297 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.132261 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw"] Oct 30 15:58:44.140308 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.140268 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-d7jfw"] Oct 30 15:58:44.180047 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.180003 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj"] Oct 30 15:58:44.185423 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.185386 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-b24e7b-openshift-default-b6dd9956f-mltkj"] Oct 30 15:58:44.663386 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.663309 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm"] Oct 30 15:58:44.664798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664765 4401 cpu_manager.go:414] "containerMap: removing container" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" Oct 30 15:58:44.664798 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664797 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" Oct 30 15:58:44.665061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664840 4401 cpu_manager.go:414] "containerMap: removing container" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" Oct 30 15:58:44.665061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664848 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" Oct 30 15:58:44.665061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664900 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" Oct 30 15:58:44.665061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664908 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" Oct 30 15:58:44.665061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664936 4401 cpu_manager.go:414] "containerMap: removing container" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" Oct 30 15:58:44.665061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.664944 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" Oct 30 15:58:44.665330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.665295 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" containerName="istio-proxy" Oct 30 15:58:44.665330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.665319 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" containerName="istio-proxy" Oct 30 15:58:44.665431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.665341 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" containerName="istio-proxy" Oct 30 15:58:44.665431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.665356 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" containerName="istio-proxy" Oct 30 15:58:44.670140 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.670115 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.685016 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.682935 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm"] Oct 30 15:58:44.713575 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.713526 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.713732 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.713672 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.713930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.713902 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsbw\" (UniqueName: \"kubernetes.io/projected/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-kube-api-access-vnsbw\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.815284 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.815174 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.815284 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.815261 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.815535 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.815443 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsbw\" (UniqueName: \"kubernetes.io/projected/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-kube-api-access-vnsbw\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.816007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.815978 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.817814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.817787 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.842599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.842566 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsbw\" (UniqueName: \"kubernetes.io/projected/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-kube-api-access-vnsbw\") pod \"maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:44.947757 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.947716 4401 generic.go:356] "Generic (PLEG): container finished" podID="dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" containerID="bdf1680fcc9a7f04c8abfc7a0487c3912c0f0ee5df64c432fb3dff1680fd5fa2" exitCode=0 Oct 30 15:58:44.948016 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.947801 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" event={"ID":"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37","Type":"ContainerDied","Data":"bdf1680fcc9a7f04c8abfc7a0487c3912c0f0ee5df64c432fb3dff1680fd5fa2"} Oct 30 15:58:44.989279 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:44.989243 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:45.307652 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:45.307620 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaadbed6_ccc3_4b9b_8812_2aa80cd9eb00.slice/crio-f660d8e80889b5f66d0f7a56f95490975a530eb7064e8c28b932692b77007563 WatchSource:0}: Error finding container f660d8e80889b5f66d0f7a56f95490975a530eb7064e8c28b932692b77007563: Status 404 returned error can't find the container with id f660d8e80889b5f66d0f7a56f95490975a530eb7064e8c28b932692b77007563 Oct 30 15:58:45.309403 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:45.309372 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm"] Oct 30 15:58:45.429530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:45.429494 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35eca30d-cb5e-417b-ae35-6faefa966211" path="/var/lib/kubelet/pods/35eca30d-cb5e-417b-ae35-6faefa966211/volumes" Oct 30 15:58:45.430174 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:45.430153 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b4e65e-4696-4b8d-a5a9-f6ab671149e8" path="/var/lib/kubelet/pods/65b4e65e-4696-4b8d-a5a9-f6ab671149e8/volumes" Oct 30 15:58:45.430678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:45.430661 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e3faa20-1ade-4a35-812b-997c25755b10" path="/var/lib/kubelet/pods/7e3faa20-1ade-4a35-812b-997c25755b10/volumes" Oct 30 15:58:45.431488 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:45.431472 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf8176a4-c357-45be-b324-ccbb8c53ee36" path="/var/lib/kubelet/pods/cf8176a4-c357-45be-b324-ccbb8c53ee36/volumes" Oct 30 15:58:45.962283 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:45.962182 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" event={"ID":"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00","Type":"ContainerStarted","Data":"ed196a7ca06e2cc6f9a7453a825ed4d90d0c876e04b3db76d885e73e11503aa8"} Oct 30 15:58:45.962283 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:45.962248 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" event={"ID":"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00","Type":"ContainerStarted","Data":"f660d8e80889b5f66d0f7a56f95490975a530eb7064e8c28b932692b77007563"} Oct 30 15:58:46.011762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.011544 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" podStartSLOduration=2.011528442 podStartE2EDuration="2.011528442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:46.006279467 +0000 UTC m=+1234.705552635" watchObservedRunningTime="2026-10-30 15:58:46.011528442 +0000 UTC m=+1234.710801598" Oct 30 15:58:46.125191 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.125163 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:46.183959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.183919 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-xks-ca\") pod \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " Oct 30 15:58:46.183959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.183964 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-kube-api-access-gh7c7\" (UniqueName: \"kubernetes.io/projected/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-kube-api-access-gh7c7\") pod \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " Oct 30 15:58:46.184206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.184063 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-openshift-ca\") pod \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\" (UID: \"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37\") " Oct 30 15:58:46.184529 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.184502 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-openshift-ca" pod "dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" (UID: "dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:46.186223 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.186193 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-xks-ca" pod "dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" (UID: "dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:46.186789 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.186770 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-kube-api-access-gh7c7" pod "dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" (UID: "dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37"). InnerVolumeSpecName "kube-api-access-gh7c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:46.320459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.319911 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:46.320459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.319956 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:46.320459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.319970 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh7c7\" (UniqueName: \"kubernetes.io/projected/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37-kube-api-access-gh7c7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:46.974119 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.974084 4401 generic.go:356] "Generic (PLEG): container finished" podID="faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" containerID="ed196a7ca06e2cc6f9a7453a825ed4d90d0c876e04b3db76d885e73e11503aa8" exitCode=0 Oct 30 15:58:46.974548 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.974181 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" event={"ID":"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00","Type":"ContainerDied","Data":"ed196a7ca06e2cc6f9a7453a825ed4d90d0c876e04b3db76d885e73e11503aa8"} Oct 30 15:58:46.975817 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.975793 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" event={"ID":"dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37","Type":"ContainerDied","Data":"49ab0ab79c9207a36c11e4f3b633dc694a7fd61164345a1af0424cde16386ede"} Oct 30 15:58:46.975966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.975817 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb" Oct 30 15:58:46.975966 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:46.975867 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ab0ab79c9207a36c11e4f3b633dc694a7fd61164345a1af0424cde16386ede" Oct 30 15:58:48.154182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.154153 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:48.243359 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.243248 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-kube-api-access-vnsbw\" (UniqueName: \"kubernetes.io/projected/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-kube-api-access-vnsbw\") pod \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " Oct 30 15:58:48.243539 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.243363 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-xks-ca\") pod \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " Oct 30 15:58:48.243539 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.243481 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-openshift-ca\") pod \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\" (UID: \"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00\") " Oct 30 15:58:48.243888 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.243861 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-openshift-ca" pod "faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" (UID: "faadbed6-ccc3-4b9b-8812-2aa80cd9eb00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:58:48.244006 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.243989 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:48.245746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.245712 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-xks-ca" pod "faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" (UID: "faadbed6-ccc3-4b9b-8812-2aa80cd9eb00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:58:48.246071 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.246048 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-kube-api-access-vnsbw" pod "faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" (UID: "faadbed6-ccc3-4b9b-8812-2aa80cd9eb00"). InnerVolumeSpecName "kube-api-access-vnsbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:58:48.345849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.345316 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:48.345849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.345360 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsbw\" (UniqueName: \"kubernetes.io/projected/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00-kube-api-access-vnsbw\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:58:48.995553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.995521 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" Oct 30 15:58:48.995553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.995545 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm" event={"ID":"faadbed6-ccc3-4b9b-8812-2aa80cd9eb00","Type":"ContainerDied","Data":"f660d8e80889b5f66d0f7a56f95490975a530eb7064e8c28b932692b77007563"} Oct 30 15:58:48.995915 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:48.995594 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f660d8e80889b5f66d0f7a56f95490975a530eb7064e8c28b932692b77007563" Oct 30 15:58:49.040694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:49.040655 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh"] Oct 30 15:58:49.068046 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:49.068008 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf"] Oct 30 15:58:49.451609 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:49.451571 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4"] Oct 30 15:58:49.458372 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:49.458337 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp"] Oct 30 15:58:49.479096 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:49.479055 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985"] Oct 30 15:58:51.970036 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:58:51.969999 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:58:52.357324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:52.357284 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q"] Oct 30 15:58:52.384449 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:52.384411 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j"] Oct 30 15:58:54.708932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:54.708885 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56"] Oct 30 15:58:55.414268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:55.414227 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp"] Oct 30 15:58:56.874053 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.874018 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6"] Oct 30 15:58:56.874649 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.874634 4401 cpu_manager.go:414] "containerMap: removing container" podUID="dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" containerName="revoke-keys" Oct 30 15:58:56.874694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.874650 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" containerName="revoke-keys" Oct 30 15:58:56.874694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.874669 4401 cpu_manager.go:414] "containerMap: removing container" podUID="faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" containerName="revoke-keys" Oct 30 15:58:56.874694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.874676 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" containerName="revoke-keys" Oct 30 15:58:56.874791 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.874781 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" containerName="revoke-keys" Oct 30 15:58:56.874791 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.874790 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" containerName="revoke-keys" Oct 30 15:58:56.878528 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.878510 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:56.896860 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.896810 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6"] Oct 30 15:58:56.904478 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.904439 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:58:56.905211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.905164 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:58:56.911024 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.910994 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:58:56.911177 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.911047 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:58:56.930316 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.930278 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:56.930527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.930327 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458lr\" (UniqueName: \"kubernetes.io/projected/9da206b1-06a1-44fa-9a61-29211b75cc57-kube-api-access-458lr\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:56.930527 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.930489 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:56.962932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.962895 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk"] Oct 30 15:58:56.967565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.967542 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:56.985094 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:56.985048 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk"] Oct 30 15:58:57.031580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.031538 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:57.031793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.031593 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458lr\" (UniqueName: \"kubernetes.io/projected/9da206b1-06a1-44fa-9a61-29211b75cc57-kube-api-access-458lr\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:57.031793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.031653 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.031793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.031732 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.032048 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.031841 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:57.032048 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.031951 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8wl\" (UniqueName: \"kubernetes.io/projected/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-kube-api-access-wt8wl\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.032378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.032353 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:57.034276 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.034243 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:57.053174 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.053133 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx"] Oct 30 15:58:57.058154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.058125 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.061500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.061463 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458lr\" (UniqueName: \"kubernetes.io/projected/9da206b1-06a1-44fa-9a61-29211b75cc57-kube-api-access-458lr\") pod \"maas-api-revoke-sub-ae73c45b-l9wk6\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:57.081137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.081103 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx"] Oct 30 15:58:57.133520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.133420 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.133682 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.133569 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.133682 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.133635 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8wl\" (UniqueName: \"kubernetes.io/projected/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-kube-api-access-wt8wl\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.133779 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.133686 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.133779 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.133706 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.133779 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.133755 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g52v\" (UniqueName: \"kubernetes.io/projected/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-kube-api-access-7g52v\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.134282 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.134262 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.135975 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.135954 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.159325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.159294 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8wl\" (UniqueName: \"kubernetes.io/projected/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-kube-api-access-wt8wl\") pod \"maas-api-revoke-sub-194f8cfb-dnqwk\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.194844 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.194801 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:58:57.235211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.235160 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.235211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.235202 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.235521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.235247 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g52v\" (UniqueName: \"kubernetes.io/projected/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-kube-api-access-7g52v\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.236014 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.235988 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.237679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.237660 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.275254 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.275215 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g52v\" (UniqueName: \"kubernetes.io/projected/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-kube-api-access-7g52v\") pod \"maas-api-revoke-sub-c878956b-scznx\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.285088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.285025 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm"] Oct 30 15:58:57.286783 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.286710 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:58:57.291599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.291579 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.333666 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.333618 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7"] Oct 30 15:58:57.338353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338317 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338410 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338461 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nps\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-kube-api-access-w7nps\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338526 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb54a07-e06c-4575-8d85-cf9bb779283f-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338553 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338586 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338616 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338681 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.338795 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.338740 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.339593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.339545 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-040f9a-openshift-default-dockercfg-r5zts\"" Oct 30 15:58:57.340701 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.340671 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.351130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.351094 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm"] Oct 30 15:58:57.362211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.359432 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" podUID="66b3a931-864d-44fe-99d6-813f944c8a3b" containerName="payload-processing" containerID="cri-o://b82affa4b62a9bb0042f60b084677bde3228817daa1dd419e402e28f386fa4d4" gracePeriod=25 Oct 30 15:58:57.387503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.387428 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" podUID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" containerName="payload-pre-processing" containerID="cri-o://e1a96dc874ec5970db728e073417a84c32bf791d371552af821715939ade6932" gracePeriod=25 Oct 30 15:58:57.391151 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.391111 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:58:57.397411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.397374 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7"] Oct 30 15:58:57.440638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.440599 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.440999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.440971 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.441179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441157 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlrq\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-kube-api-access-4tlrq\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.441333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441316 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.441485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441469 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.441609 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441593 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.441699 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441679 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.441770 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441012 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.441884 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441204 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.442017 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.441995 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.442149 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.442132 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.442278 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.442261 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.442405 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.442388 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nps\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-kube-api-access-w7nps\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.442521 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.442506 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.442639 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.442623 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.442835 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.442578 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.442960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.442862 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb54a07-e06c-4575-8d85-cf9bb779283f-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.443095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.443079 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.443233 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.443217 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.443419 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.443402 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.443574 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.443557 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.443723 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.443707 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.444530 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.443567 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb54a07-e06c-4575-8d85-cf9bb779283f-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.445899 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.445842 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.447435 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.447404 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.481061 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.474655 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nps\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-kube-api-access-w7nps\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.484880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.481736 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.494063 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.494014 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g"] Oct 30 15:58:57.510494 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.510426 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.524761 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.524729 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6"] Oct 30 15:58:57.541367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.541335 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g"] Oct 30 15:58:57.545381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545248 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545327 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545352 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnq9\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-kube-api-access-jwnq9\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545417 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545486 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.545603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545532 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.545603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545568 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545603 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545597 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545656 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545760 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.545883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545814 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlrq\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-kube-api-access-4tlrq\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.545883 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545868 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.546095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545930 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.546095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.545975 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.546095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.546023 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.546095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.546069 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.546301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.546102 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.546301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.546147 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.547106 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.546749 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.547221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.547103 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-credential-socket\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.547221 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.547147 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-certs\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.547353 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.547245 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istiod-ca-cert\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.547415 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.547386 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-data\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.548878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.548858 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-envoy\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.549934 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.549910 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-podinfo\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.571882 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.571842 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-token\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.572034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.571897 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlrq\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-kube-api-access-4tlrq\") pod \"e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.611636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.611598 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663037 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663118 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663283 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663317 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnq9\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-kube-api-access-jwnq9\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663345 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663413 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663440 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663503 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.663619 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.670097 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.664232 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.674660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.673269 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.674660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.673514 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.674660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.674151 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.674660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.674592 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.693856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.692932 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:57.693856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.693247 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk"] Oct 30 15:58:57.747557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.747518 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.748219 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.748195 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.748417 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.748387 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnq9\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-kube-api-access-jwnq9\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.748607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.748588 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:57.750260 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:57.750227 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31ffe21_4522_4e15_bb9e_e9e0bfe13225.slice/crio-81567aeb070658e9ab85aaefa2361ea4a0a9d537230da89bd1daa70777b9c168 WatchSource:0}: Error finding container 81567aeb070658e9ab85aaefa2361ea4a0a9d537230da89bd1daa70777b9c168: Status 404 returned error can't find the container with id 81567aeb070658e9ab85aaefa2361ea4a0a9d537230da89bd1daa70777b9c168 Oct 30 15:58:57.767787 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.767645 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx"] Oct 30 15:58:57.767878 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:57.767840 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ddbd7c6_7d5a_4482_ba7b_8cd4f7010675.slice/crio-976c647ff94018eff5db242142141caee01415489a94f4ff4df427f5c346d474 WatchSource:0}: Error finding container 976c647ff94018eff5db242142141caee01415489a94f4ff4df427f5c346d474: Status 404 returned error can't find the container with id 976c647ff94018eff5db242142141caee01415489a94f4ff4df427f5c346d474 Oct 30 15:58:57.843594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:57.843564 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:58.083703 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:58.083674 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb54a07_e06c_4575_8d85_cf9bb779283f.slice/crio-b5efb8cf9f4d9ec208cd15ee28d015f813249aa01841f0899060c9f7ddd5a955 WatchSource:0}: Error finding container b5efb8cf9f4d9ec208cd15ee28d015f813249aa01841f0899060c9f7ddd5a955: Status 404 returned error can't find the container with id b5efb8cf9f4d9ec208cd15ee28d015f813249aa01841f0899060c9f7ddd5a955 Oct 30 15:58:58.086104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.085727 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm"] Oct 30 15:58:58.086939 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.086901 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.087035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.086971 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.087035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.087002 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.096837 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.096794 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7"] Oct 30 15:58:58.102062 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:58:58.102026 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d546ec3_604a_45bc_9a22_5a3e67f069e8.slice/crio-763ca8d87eb00d4b16893dc302664721cf6d5b05045869da1e6996f7e734ea9a WatchSource:0}: Error finding container 763ca8d87eb00d4b16893dc302664721cf6d5b05045869da1e6996f7e734ea9a: Status 404 returned error can't find the container with id 763ca8d87eb00d4b16893dc302664721cf6d5b05045869da1e6996f7e734ea9a Oct 30 15:58:58.105436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.105392 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.105541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.105489 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.105541 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.105531 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.166220 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.166183 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" event={"ID":"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675","Type":"ContainerStarted","Data":"976c647ff94018eff5db242142141caee01415489a94f4ff4df427f5c346d474"} Oct 30 15:58:58.168360 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.168322 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" event={"ID":"9da206b1-06a1-44fa-9a61-29211b75cc57","Type":"ContainerStarted","Data":"8481c9b0b892b5744fffaf44d018d87b7fd56976b423f12f5b91606f25ca8551"} Oct 30 15:58:58.168519 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.168388 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" event={"ID":"9da206b1-06a1-44fa-9a61-29211b75cc57","Type":"ContainerStarted","Data":"c13b45efd8ef6ca40a720a8ee271374b8dbca72e1fdd69ea45c8ccbb966a4e15"} Oct 30 15:58:58.170522 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.170494 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" event={"ID":"f31ffe21-4522-4e15-bb9e-e9e0bfe13225","Type":"ContainerStarted","Data":"81567aeb070658e9ab85aaefa2361ea4a0a9d537230da89bd1daa70777b9c168"} Oct 30 15:58:58.172346 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.172315 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" event={"ID":"0bb54a07-e06c-4575-8d85-cf9bb779283f","Type":"ContainerStarted","Data":"b5efb8cf9f4d9ec208cd15ee28d015f813249aa01841f0899060c9f7ddd5a955"} Oct 30 15:58:58.174158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.174122 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" event={"ID":"2d546ec3-604a-45bc-9a22-5a3e67f069e8","Type":"ContainerStarted","Data":"763ca8d87eb00d4b16893dc302664721cf6d5b05045869da1e6996f7e734ea9a"} Oct 30 15:58:58.219627 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.219552 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" podStartSLOduration=2.219529096 podStartE2EDuration="2.219529096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:58.214456739 +0000 UTC m=+1246.913729899" watchObservedRunningTime="2026-10-30 15:58:58.219529096 +0000 UTC m=+1246.918802254" Oct 30 15:58:58.233578 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.233402 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g"] Oct 30 15:58:58.235840 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.235793 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.235961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.235902 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.235961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.235934 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:58:58.262979 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:58.262923 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" podStartSLOduration=2.262907132 podStartE2EDuration="2.262907132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:58.26093546 +0000 UTC m=+1246.960208616" watchObservedRunningTime="2026-10-30 15:58:58.262907132 +0000 UTC m=+1246.962180289" Oct 30 15:58:59.232247 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.232157 4401 generic.go:356] "Generic (PLEG): container finished" podID="9da206b1-06a1-44fa-9a61-29211b75cc57" containerID="8481c9b0b892b5744fffaf44d018d87b7fd56976b423f12f5b91606f25ca8551" exitCode=0 Oct 30 15:58:59.232624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.232264 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" event={"ID":"9da206b1-06a1-44fa-9a61-29211b75cc57","Type":"ContainerDied","Data":"8481c9b0b892b5744fffaf44d018d87b7fd56976b423f12f5b91606f25ca8551"} Oct 30 15:58:59.234584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.234551 4401 generic.go:356] "Generic (PLEG): container finished" podID="f31ffe21-4522-4e15-bb9e-e9e0bfe13225" containerID="1eccffef68629524c9a8cdf65879c3d9397759c6f6cae5340012cbe854aff176" exitCode=0 Oct 30 15:58:59.234724 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.234669 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" event={"ID":"f31ffe21-4522-4e15-bb9e-e9e0bfe13225","Type":"ContainerDied","Data":"1eccffef68629524c9a8cdf65879c3d9397759c6f6cae5340012cbe854aff176"} Oct 30 15:58:59.238108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.238080 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" event={"ID":"0bb54a07-e06c-4575-8d85-cf9bb779283f","Type":"ContainerStarted","Data":"00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491"} Oct 30 15:58:59.240360 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.240321 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" event={"ID":"2bfdc81f-a987-4bad-903d-ece8c84f97e5","Type":"ContainerStarted","Data":"74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d"} Oct 30 15:58:59.240485 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.240373 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" event={"ID":"2bfdc81f-a987-4bad-903d-ece8c84f97e5","Type":"ContainerStarted","Data":"ba953056828362f6d99c6b7ba1058326046201015460ba44c18475b4fac2d7ce"} Oct 30 15:58:59.242324 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.242292 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" event={"ID":"2d546ec3-604a-45bc-9a22-5a3e67f069e8","Type":"ContainerStarted","Data":"1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327"} Oct 30 15:58:59.244820 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.244794 4401 generic.go:356] "Generic (PLEG): container finished" podID="9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" containerID="a5fcce886fbbc66bcc932edf908f18e12060472536b00b3c072b7c1474a7d641" exitCode=0 Oct 30 15:58:59.244947 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.244841 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" event={"ID":"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675","Type":"ContainerDied","Data":"a5fcce886fbbc66bcc932edf908f18e12060472536b00b3c072b7c1474a7d641"} Oct 30 15:58:59.530044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.523361 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podStartSLOduration=2.508364113 podStartE2EDuration="2.508364113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:59.373124577 +0000 UTC m=+1248.072397737" watchObservedRunningTime="2026-10-30 15:58:59.508364113 +0000 UTC m=+1248.207637292" Oct 30 15:58:59.611909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.587707 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podStartSLOduration=2.587685678 podStartE2EDuration="2.587685678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:59.470426612 +0000 UTC m=+1248.169699769" watchObservedRunningTime="2026-10-30 15:58:59.587685678 +0000 UTC m=+1248.286958836" Oct 30 15:58:59.636290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.632039 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:58:59.638601 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.638561 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 15:58:59.638954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.638911 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 15:58:59.699194 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.699156 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:58:59.699558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.699533 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 15:58:59.699704 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.699680 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 15:58:59.716492 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.716439 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" podUID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" containerName="payload-processing" containerID="cri-o://e113cb88be99df19ad677cb27ee659d7e72977caae97ff990d3627db134e729e" gracePeriod=25 Oct 30 15:58:59.822656 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.822494 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podStartSLOduration=2.822469206 podStartE2EDuration="2.822469206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:58:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:58:59.743553046 +0000 UTC m=+1248.442826234" watchObservedRunningTime="2026-10-30 15:58:59.822469206 +0000 UTC m=+1248.521742363" Oct 30 15:58:59.843851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.843799 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:58:59.845394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.845350 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:58:59.845522 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:58:59.845424 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:59:00.280849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.280324 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:59:00.444990 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.439553 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerName="payload-processing" containerID="cri-o://7a07aa5051c09b22ff7780ef870534aa1cf317acd5db1c1df64cca1680adf417" gracePeriod=25 Oct 30 15:59:00.619888 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.619371 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 15:59:00.619888 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.619457 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 15:59:00.744548 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.744509 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 15:59:00.744746 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.744569 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 15:59:00.845785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.844394 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:59:00.845785 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:00.844456 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:59:01.045217 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.045182 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:59:01.391605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.366104 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-kube-api-access-wt8wl\" (UniqueName: \"kubernetes.io/projected/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-kube-api-access-wt8wl\") pod \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " Oct 30 15:59:01.391605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.366251 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-openshift-ca\") pod \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " Oct 30 15:59:01.391605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.366287 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-xks-ca\") pod \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\" (UID: \"f31ffe21-4522-4e15-bb9e-e9e0bfe13225\") " Oct 30 15:59:01.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.523263 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-kube-api-access-wt8wl" pod "f31ffe21-4522-4e15-bb9e-e9e0bfe13225" (UID: "f31ffe21-4522-4e15-bb9e-e9e0bfe13225"). InnerVolumeSpecName "kube-api-access-wt8wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:01.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.528966 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" podUID="66b3a931-864d-44fe-99d6-813f944c8a3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:01.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.533351 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-openshift-ca" pod "f31ffe21-4522-4e15-bb9e-e9e0bfe13225" (UID: "f31ffe21-4522-4e15-bb9e-e9e0bfe13225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:01.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.535751 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.539852 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.535783 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8wl\" (UniqueName: \"kubernetes.io/projected/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-kube-api-access-wt8wl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.542336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.542282 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-xks-ca" pod "f31ffe21-4522-4e15-bb9e-e9e0bfe13225" (UID: "f31ffe21-4522-4e15-bb9e-e9e0bfe13225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:01.577850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.567224 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:59:01.627796 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.627750 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" event={"ID":"f31ffe21-4522-4e15-bb9e-e9e0bfe13225","Type":"ContainerDied","Data":"81567aeb070658e9ab85aaefa2361ea4a0a9d537230da89bd1daa70777b9c168"} Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.632918 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81567aeb070658e9ab85aaefa2361ea4a0a9d537230da89bd1daa70777b9c168" Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.632453 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.632985 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.631288 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk" Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.636377 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-xks-ca\") pod \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.636497 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-kube-api-access-7g52v\" (UniqueName: \"kubernetes.io/projected/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-kube-api-access-7g52v\") pod \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.636697 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-openshift-ca\") pod \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\" (UID: \"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675\") " Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.637250 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f31ffe21-4522-4e15-bb9e-e9e0bfe13225-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.638459 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.638177 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-openshift-ca" pod "9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" (UID: "9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:01.640427 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.640365 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" event={"ID":"9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675","Type":"ContainerDied","Data":"976c647ff94018eff5db242142141caee01415489a94f4ff4df427f5c346d474"} Oct 30 15:59:01.640586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.640440 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="976c647ff94018eff5db242142141caee01415489a94f4ff4df427f5c346d474" Oct 30 15:59:01.640586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.640472 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-kube-api-access-7g52v" pod "9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" (UID: "9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675"). InnerVolumeSpecName "kube-api-access-7g52v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:01.640586 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.640548 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx" Oct 30 15:59:01.643088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.643016 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:59:01.643544 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.643512 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-xks-ca" pod "9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" (UID: "9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:01.645027 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.644998 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" event={"ID":"9da206b1-06a1-44fa-9a61-29211b75cc57","Type":"ContainerDied","Data":"c13b45efd8ef6ca40a720a8ee271374b8dbca72e1fdd69ea45c8ccbb966a4e15"} Oct 30 15:59:01.645158 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.645076 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c13b45efd8ef6ca40a720a8ee271374b8dbca72e1fdd69ea45c8ccbb966a4e15" Oct 30 15:59:01.733955 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.733918 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 15:59:01.734235 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.734205 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 15:59:01.737719 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.737681 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da206b1-06a1-44fa-9a61-29211b75cc57-kube-api-access-458lr\" (UniqueName: \"kubernetes.io/projected/9da206b1-06a1-44fa-9a61-29211b75cc57-kube-api-access-458lr\") pod \"9da206b1-06a1-44fa-9a61-29211b75cc57\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " Oct 30 15:59:01.737920 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.737864 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-openshift-ca\") pod \"9da206b1-06a1-44fa-9a61-29211b75cc57\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " Oct 30 15:59:01.738033 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.738014 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-xks-ca\") pod \"9da206b1-06a1-44fa-9a61-29211b75cc57\" (UID: \"9da206b1-06a1-44fa-9a61-29211b75cc57\") " Oct 30 15:59:01.738859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.738811 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.739928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.739897 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g52v\" (UniqueName: \"kubernetes.io/projected/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-kube-api-access-7g52v\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.740090 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.739933 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.740698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.740280 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-openshift-ca" pod "9da206b1-06a1-44fa-9a61-29211b75cc57" (UID: "9da206b1-06a1-44fa-9a61-29211b75cc57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:01.742669 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.742637 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da206b1-06a1-44fa-9a61-29211b75cc57-kube-api-access-458lr" pod "9da206b1-06a1-44fa-9a61-29211b75cc57" (UID: "9da206b1-06a1-44fa-9a61-29211b75cc57"). InnerVolumeSpecName "kube-api-access-458lr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:01.743012 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.742975 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-xks-ca" pod "9da206b1-06a1-44fa-9a61-29211b75cc57" (UID: "9da206b1-06a1-44fa-9a61-29211b75cc57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:01.841381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.840935 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.841381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.840968 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da206b1-06a1-44fa-9a61-29211b75cc57-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.841381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.840978 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458lr\" (UniqueName: \"kubernetes.io/projected/9da206b1-06a1-44fa-9a61-29211b75cc57-kube-api-access-458lr\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:01.844630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.844595 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:59:01.844782 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:01.844654 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:59:02.135243 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.135200 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:02.135243 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.135228 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" podUID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:02.622555 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.622510 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 15:59:02.623620 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.623580 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 15:59:02.729856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.711395 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6" Oct 30 15:59:02.729856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.722985 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 15:59:02.729856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.723050 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 15:59:02.844567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.844530 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:59:02.844744 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:02.844586 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:59:03.164118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:03.164065 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" podUID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:03.615853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:03.615260 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 15:59:03.615853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:03.615330 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 15:59:03.695850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:03.695301 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 15:59:03.695850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:03.695366 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 15:59:03.844182 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:03.844143 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:59:03.844379 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:03.844203 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:59:04.072909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.072228 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" containerName="main" containerID="cri-o://6671653069cc6dc6c73d6875df5ebe8b32f943438926ad68974e3ec984ade67a" gracePeriod=45 Oct 30 15:59:04.103173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.094127 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" containerID="cri-o://5c94bba59ffdad9cb6813c726f92ab8b145d560b9073376f0c0cad68d8363e91" gracePeriod=45 Oct 30 15:59:04.230595 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.230554 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused" Oct 30 15:59:04.497526 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.493027 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" containerID="cri-o://a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230" gracePeriod=45 Oct 30 15:59:04.537383 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.518094 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" containerID="cri-o://9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566" gracePeriod=45 Oct 30 15:59:04.537383 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.533127 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" containerID="cri-o://eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b" gracePeriod=45 Oct 30 15:59:04.695960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.614255 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 15:59:04.695960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.614318 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 15:59:04.695960 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.644681 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": dial tcp 10.133.0.67:8000: connect: connection refused" Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.703550 4401 patch_prober.go:29] interesting pod/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.703605 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.843516 4401 generic.go:356] "Generic (PLEG): container finished" podID="a87bdf18-4e99-49df-aac9-1d499413de55" containerID="6671653069cc6dc6c73d6875df5ebe8b32f943438926ad68974e3ec984ade67a" exitCode=0 Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.843629 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" event={"ID":"a87bdf18-4e99-49df-aac9-1d499413de55","Type":"ContainerDied","Data":"6671653069cc6dc6c73d6875df5ebe8b32f943438926ad68974e3ec984ade67a"} Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.844015 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.844069 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.848416 4401 generic.go:356] "Generic (PLEG): container finished" podID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerID="5c94bba59ffdad9cb6813c726f92ab8b145d560b9073376f0c0cad68d8363e91" exitCode=0 Oct 30 15:59:04.855436 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.848768 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" event={"ID":"6dcf71f9-764f-4251-bf81-bb3222b85808","Type":"ContainerDied","Data":"5c94bba59ffdad9cb6813c726f92ab8b145d560b9073376f0c0cad68d8363e91"} Oct 30 15:59:04.879950 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.858924 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": dial tcp 10.133.0.124:8000: connect: connection refused" Oct 30 15:59:04.926421 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.911938 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.140:8000/ready\": dial tcp 10.133.0.140:8000: connect: connection refused" Oct 30 15:59:04.926421 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:04.914044 4401 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f10fad_1159_4cc9_88b7_9da9159371a1.slice/crio-eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f10fad_1159_4cc9_88b7_9da9159371a1.slice/crio-conmon-eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2f5c3db_a820_403d_a478_91c0ebe886c0.slice/crio-conmon-9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:59:04.981891 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:04.978338 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:59:05.073125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073088 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-home\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-home\") pod \"a87bdf18-4e99-49df-aac9-1d499413de55\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " Oct 30 15:59:05.073281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073175 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-tmp-dir\") pod \"a87bdf18-4e99-49df-aac9-1d499413de55\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " Oct 30 15:59:05.073281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073259 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a87bdf18-4e99-49df-aac9-1d499413de55-kube-api-access-mcscx\" (UniqueName: \"kubernetes.io/projected/a87bdf18-4e99-49df-aac9-1d499413de55-kube-api-access-mcscx\") pod \"a87bdf18-4e99-49df-aac9-1d499413de55\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " Oct 30 15:59:05.073401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073339 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-model-cache\") pod \"a87bdf18-4e99-49df-aac9-1d499413de55\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " Oct 30 15:59:05.073401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073381 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-home" pod "a87bdf18-4e99-49df-aac9-1d499413de55" (UID: "a87bdf18-4e99-49df-aac9-1d499413de55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.073499 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073424 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a87bdf18-4e99-49df-aac9-1d499413de55-tls-certs\" (UniqueName: \"kubernetes.io/secret/a87bdf18-4e99-49df-aac9-1d499413de55-tls-certs\") pod \"a87bdf18-4e99-49df-aac9-1d499413de55\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " Oct 30 15:59:05.073499 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073463 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-dshm\") pod \"a87bdf18-4e99-49df-aac9-1d499413de55\" (UID: \"a87bdf18-4e99-49df-aac9-1d499413de55\") " Oct 30 15:59:05.074474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.073922 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-model-cache" pod "a87bdf18-4e99-49df-aac9-1d499413de55" (UID: "a87bdf18-4e99-49df-aac9-1d499413de55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.074474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.074120 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.074474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.074139 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.074711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.074507 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-tmp-dir" pod "a87bdf18-4e99-49df-aac9-1d499413de55" (UID: "a87bdf18-4e99-49df-aac9-1d499413de55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.078522 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.078472 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87bdf18-4e99-49df-aac9-1d499413de55-tls-certs" pod "a87bdf18-4e99-49df-aac9-1d499413de55" (UID: "a87bdf18-4e99-49df-aac9-1d499413de55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:05.078670 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.078612 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87bdf18-4e99-49df-aac9-1d499413de55-kube-api-access-mcscx" pod "a87bdf18-4e99-49df-aac9-1d499413de55" (UID: "a87bdf18-4e99-49df-aac9-1d499413de55"). InnerVolumeSpecName "kube-api-access-mcscx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:05.079956 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.079914 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-dshm" pod "a87bdf18-4e99-49df-aac9-1d499413de55" (UID: "a87bdf18-4e99-49df-aac9-1d499413de55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.084159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.084130 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:59:05.191301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.190807 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.191301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.190866 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87bdf18-4e99-49df-aac9-1d499413de55-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.191301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.190888 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcscx\" (UniqueName: \"kubernetes.io/projected/a87bdf18-4e99-49df-aac9-1d499413de55-kube-api-access-mcscx\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.191301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.190904 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87bdf18-4e99-49df-aac9-1d499413de55-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.271329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.271289 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:59:05.278484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.278447 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:59:05.287371 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.287339 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:59:05.291612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.291560 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-model-cache\") pod \"6dcf71f9-764f-4251-bf81-bb3222b85808\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.291720 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-dshm\") pod \"6dcf71f9-764f-4251-bf81-bb3222b85808\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.292142 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-model-cache" pod "6dcf71f9-764f-4251-bf81-bb3222b85808" (UID: "6dcf71f9-764f-4251-bf81-bb3222b85808"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.292619 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-dshm\") pod \"a2f5c3db-a820-403d-a478-91c0ebe886c0\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.293061 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-dshm\") pod \"b1f10fad-1159-4cc9-88b7-9da9159371a1\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.293405 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-home\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-home\") pod \"a2f5c3db-a820-403d-a478-91c0ebe886c0\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.293697 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-home" pod "a2f5c3db-a820-403d-a478-91c0ebe886c0" (UID: "a2f5c3db-a820-403d-a478-91c0ebe886c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.293862 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-tmp-dir\") pod \"a2f5c3db-a820-403d-a478-91c0ebe886c0\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.293911 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tls-certs\") pod \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.293953 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-model-cache\") pod \"a2f5c3db-a820-403d-a478-91c0ebe886c0\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294005 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2f5c3db-a820-403d-a478-91c0ebe886c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f5c3db-a820-403d-a478-91c0ebe886c0-tls-certs\") pod \"a2f5c3db-a820-403d-a478-91c0ebe886c0\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294091 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dcf71f9-764f-4251-bf81-bb3222b85808-tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf71f9-764f-4251-bf81-bb3222b85808-tls-certs\") pod \"6dcf71f9-764f-4251-bf81-bb3222b85808\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294130 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dcf71f9-764f-4251-bf81-bb3222b85808-kube-api-access-t99gj\" (UniqueName: \"kubernetes.io/projected/6dcf71f9-764f-4251-bf81-bb3222b85808-kube-api-access-t99gj\") pod \"6dcf71f9-764f-4251-bf81-bb3222b85808\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294162 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2f5c3db-a820-403d-a478-91c0ebe886c0-kube-api-access-lcxd5\" (UniqueName: \"kubernetes.io/projected/a2f5c3db-a820-403d-a478-91c0ebe886c0-kube-api-access-lcxd5\") pod \"a2f5c3db-a820-403d-a478-91c0ebe886c0\" (UID: \"a2f5c3db-a820-403d-a478-91c0ebe886c0\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294225 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-model-cache\") pod \"b1f10fad-1159-4cc9-88b7-9da9159371a1\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294269 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-tmp-dir\") pod \"6dcf71f9-764f-4251-bf81-bb3222b85808\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294305 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-dshm\") pod \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294338 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-home\") pod \"6dcf71f9-764f-4251-bf81-bb3222b85808\" (UID: \"6dcf71f9-764f-4251-bf81-bb3222b85808\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294373 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-model-cache\") pod \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294437 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-kube-api-access-5cplp\" (UniqueName: \"kubernetes.io/projected/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-kube-api-access-5cplp\") pod \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294488 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f10fad-1159-4cc9-88b7-9da9159371a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f10fad-1159-4cc9-88b7-9da9159371a1-tls-certs\") pod \"b1f10fad-1159-4cc9-88b7-9da9159371a1\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294521 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-home\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-home\") pod \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294584 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tmp-dir\") pod \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\" (UID: \"a129bf09-98a5-49c7-a89a-a4fda9f17ee3\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.294631 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f10fad-1159-4cc9-88b7-9da9159371a1-kube-api-access-blkx9\" (UniqueName: \"kubernetes.io/projected/b1f10fad-1159-4cc9-88b7-9da9159371a1-kube-api-access-blkx9\") pod \"b1f10fad-1159-4cc9-88b7-9da9159371a1\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.295034 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-dshm" pod "6dcf71f9-764f-4251-bf81-bb3222b85808" (UID: "6dcf71f9-764f-4251-bf81-bb3222b85808"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.295292 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-model-cache" pod "b1f10fad-1159-4cc9-88b7-9da9159371a1" (UID: "b1f10fad-1159-4cc9-88b7-9da9159371a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.295510 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-model-cache" pod "a129bf09-98a5-49c7-a89a-a4fda9f17ee3" (UID: "a129bf09-98a5-49c7-a89a-a4fda9f17ee3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.295700 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-dshm" pod "b1f10fad-1159-4cc9-88b7-9da9159371a1" (UID: "b1f10fad-1159-4cc9-88b7-9da9159371a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.295872 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-tmp-dir" pod "6dcf71f9-764f-4251-bf81-bb3222b85808" (UID: "6dcf71f9-764f-4251-bf81-bb3222b85808"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.296148 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-model-cache" pod "a2f5c3db-a820-403d-a478-91c0ebe886c0" (UID: "a2f5c3db-a820-403d-a478-91c0ebe886c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.298370 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-tmp-dir" pod "a2f5c3db-a820-403d-a478-91c0ebe886c0" (UID: "a2f5c3db-a820-403d-a478-91c0ebe886c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.298578 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-home" pod "6dcf71f9-764f-4251-bf81-bb3222b85808" (UID: "6dcf71f9-764f-4251-bf81-bb3222b85808"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.298756 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tmp-dir" pod "a129bf09-98a5-49c7-a89a-a4fda9f17ee3" (UID: "a129bf09-98a5-49c7-a89a-a4fda9f17ee3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.303388 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2f5c3db-a820-403d-a478-91c0ebe886c0-tls-certs" pod "a2f5c3db-a820-403d-a478-91c0ebe886c0" (UID: "a2f5c3db-a820-403d-a478-91c0ebe886c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.303470 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-dshm" pod "a2f5c3db-a820-403d-a478-91c0ebe886c0" (UID: "a2f5c3db-a820-403d-a478-91c0ebe886c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.303553 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tls-certs" pod "a129bf09-98a5-49c7-a89a-a4fda9f17ee3" (UID: "a129bf09-98a5-49c7-a89a-a4fda9f17ee3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.304078 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-dshm" pod "a129bf09-98a5-49c7-a89a-a4fda9f17ee3" (UID: "a129bf09-98a5-49c7-a89a-a4fda9f17ee3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.304380 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2f5c3db-a820-403d-a478-91c0ebe886c0-kube-api-access-lcxd5" pod "a2f5c3db-a820-403d-a478-91c0ebe886c0" (UID: "a2f5c3db-a820-403d-a478-91c0ebe886c0"). InnerVolumeSpecName "kube-api-access-lcxd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.303932 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dcf71f9-764f-4251-bf81-bb3222b85808-tls-certs" pod "6dcf71f9-764f-4251-bf81-bb3222b85808" (UID: "6dcf71f9-764f-4251-bf81-bb3222b85808"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:05.305125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305006 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-home" pod "a129bf09-98a5-49c7-a89a-a4fda9f17ee3" (UID: "a129bf09-98a5-49c7-a89a-a4fda9f17ee3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305422 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6dcf71f9-764f-4251-bf81-bb3222b85808-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305449 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxd5\" (UniqueName: \"kubernetes.io/projected/a2f5c3db-a820-403d-a478-91c0ebe886c0-kube-api-access-lcxd5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305468 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305482 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305495 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305507 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305519 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305531 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305543 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305555 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305567 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6dcf71f9-764f-4251-bf81-bb3222b85808-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305579 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305592 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305604 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305618 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305630 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305643 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2f5c3db-a820-403d-a478-91c0ebe886c0-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.305655 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2f5c3db-a820-403d-a478-91c0ebe886c0-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.306803 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f10fad-1159-4cc9-88b7-9da9159371a1-tls-certs" pod "b1f10fad-1159-4cc9-88b7-9da9159371a1" (UID: "b1f10fad-1159-4cc9-88b7-9da9159371a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.307145 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcf71f9-764f-4251-bf81-bb3222b85808-kube-api-access-t99gj" pod "6dcf71f9-764f-4251-bf81-bb3222b85808" (UID: "6dcf71f9-764f-4251-bf81-bb3222b85808"). InnerVolumeSpecName "kube-api-access-t99gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.307264 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f10fad-1159-4cc9-88b7-9da9159371a1-kube-api-access-blkx9" pod "b1f10fad-1159-4cc9-88b7-9da9159371a1" (UID: "b1f10fad-1159-4cc9-88b7-9da9159371a1"). InnerVolumeSpecName "kube-api-access-blkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:05.307709 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.307451 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-kube-api-access-5cplp" pod "a129bf09-98a5-49c7-a89a-a4fda9f17ee3" (UID: "a129bf09-98a5-49c7-a89a-a4fda9f17ee3"). InnerVolumeSpecName "kube-api-access-5cplp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:05.406474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.406364 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-home\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-home\") pod \"b1f10fad-1159-4cc9-88b7-9da9159371a1\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " Oct 30 15:59:05.406751 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.406729 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-tmp-dir\") pod \"b1f10fad-1159-4cc9-88b7-9da9159371a1\" (UID: \"b1f10fad-1159-4cc9-88b7-9da9159371a1\") " Oct 30 15:59:05.406968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.406628 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-home" pod "b1f10fad-1159-4cc9-88b7-9da9159371a1" (UID: "b1f10fad-1159-4cc9-88b7-9da9159371a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.407091 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.406907 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-tmp-dir" pod "b1f10fad-1159-4cc9-88b7-9da9159371a1" (UID: "b1f10fad-1159-4cc9-88b7-9da9159371a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:05.407455 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.407425 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.407503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.407462 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1f10fad-1159-4cc9-88b7-9da9159371a1-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.407503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.407480 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t99gj\" (UniqueName: \"kubernetes.io/projected/6dcf71f9-764f-4251-bf81-bb3222b85808-kube-api-access-t99gj\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.407503 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.407494 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cplp\" (UniqueName: \"kubernetes.io/projected/a129bf09-98a5-49c7-a89a-a4fda9f17ee3-kube-api-access-5cplp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.407608 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.407508 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1f10fad-1159-4cc9-88b7-9da9159371a1-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.407608 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.407521 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkx9\" (UniqueName: \"kubernetes.io/projected/b1f10fad-1159-4cc9-88b7-9da9159371a1-kube-api-access-blkx9\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:05.816035 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.815937 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:59:05.816516 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.816392 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:59:05.818667 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.818633 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:59:05.845315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.845264 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:59:05.845496 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.845329 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 15:59:05.861585 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.861534 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" containerName="main" probeResult="failure" output="Get \"https://10.133.0.137:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:59:05.865117 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.865081 4401 generic.go:356] "Generic (PLEG): container finished" podID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerID="a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230" exitCode=0 Oct 30 15:59:05.865292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.865146 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" Oct 30 15:59:05.865292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.865178 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" event={"ID":"a129bf09-98a5-49c7-a89a-a4fda9f17ee3","Type":"ContainerDied","Data":"a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230"} Oct 30 15:59:05.865292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.865230 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4" event={"ID":"a129bf09-98a5-49c7-a89a-a4fda9f17ee3","Type":"ContainerDied","Data":"3dd76c8e3e24b262cb60c61e678106871555dea2500bf30b541c49b93c46abb4"} Oct 30 15:59:05.865292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.865256 4401 scope.go:122] "RemoveContainer" containerID="a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230" Oct 30 15:59:05.868129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.867191 4401 generic.go:356] "Generic (PLEG): container finished" podID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerID="eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b" exitCode=0 Oct 30 15:59:05.868129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.867293 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" Oct 30 15:59:05.868129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.867452 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" event={"ID":"b1f10fad-1159-4cc9-88b7-9da9159371a1","Type":"ContainerDied","Data":"eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b"} Oct 30 15:59:05.868129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.867484 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp" event={"ID":"b1f10fad-1159-4cc9-88b7-9da9159371a1","Type":"ContainerDied","Data":"f2cbdb966918bb8a0e78142bb98a860c231644b176a0e8b1ef07bb077a1918bb"} Oct 30 15:59:05.870616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.870590 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" Oct 30 15:59:05.870793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.870592 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh" event={"ID":"a87bdf18-4e99-49df-aac9-1d499413de55","Type":"ContainerDied","Data":"22169147668a93249f340806aaf4a5d7e8e77d9abea6d2367d967ca3c70e4369"} Oct 30 15:59:05.873274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.873247 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" Oct 30 15:59:05.873948 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.873347 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf" event={"ID":"6dcf71f9-764f-4251-bf81-bb3222b85808","Type":"ContainerDied","Data":"ff6ae24ab1e4ee42ead0fdf95165d21750920c2522ceb5f67a1ad6cf822af9a0"} Oct 30 15:59:05.876614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.876570 4401 generic.go:356] "Generic (PLEG): container finished" podID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerID="9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566" exitCode=0 Oct 30 15:59:05.877432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.877410 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" Oct 30 15:59:05.877781 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.877761 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" event={"ID":"a2f5c3db-a820-403d-a478-91c0ebe886c0","Type":"ContainerDied","Data":"9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566"} Oct 30 15:59:05.877896 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.877794 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985" event={"ID":"a2f5c3db-a820-403d-a478-91c0ebe886c0","Type":"ContainerDied","Data":"ab51106ee1322a65b670d52a1e5b750db62c8eebe102cfbdfa141efcf4617310"} Oct 30 15:59:05.884761 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.884730 4401 scope.go:122] "RemoveContainer" containerID="a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230" Oct 30 15:59:05.885129 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:05.885106 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230\": container with ID starting with a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230 not found: ID does not exist" containerID="a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230" Oct 30 15:59:05.885231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.885140 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230"} err="failed to get container status \"a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230\": rpc error: code = NotFound desc = could not find container \"a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230\": container with ID starting with a804ea8ed2c35abd9a75383946a4c6152b502aa9388f44d31618caac01051230 not found: ID does not exist" Oct 30 15:59:05.885231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.885163 4401 scope.go:122] "RemoveContainer" containerID="eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b" Oct 30 15:59:05.894596 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.894545 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:59:05.895109 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.895033 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:59:05.897452 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.897428 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:59:05.900246 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.900223 4401 scope.go:122] "RemoveContainer" containerID="eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b" Oct 30 15:59:05.900713 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:05.900685 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b\": container with ID starting with eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b not found: ID does not exist" containerID="eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b" Oct 30 15:59:05.900803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.900725 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b"} err="failed to get container status \"eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b\": rpc error: code = NotFound desc = could not find container \"eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b\": container with ID starting with eb5da615d888ea51170b837744d33705c13b7d5b09942d0efbdeb6c6640aa85b not found: ID does not exist" Oct 30 15:59:05.900803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.900752 4401 scope.go:122] "RemoveContainer" containerID="6671653069cc6dc6c73d6875df5ebe8b32f943438926ad68974e3ec984ade67a" Oct 30 15:59:05.913486 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.913460 4401 scope.go:122] "RemoveContainer" containerID="5c94bba59ffdad9cb6813c726f92ab8b145d560b9073376f0c0cad68d8363e91" Oct 30 15:59:05.927992 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.927966 4401 scope.go:122] "RemoveContainer" containerID="9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566" Oct 30 15:59:05.944402 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.944366 4401 scope.go:122] "RemoveContainer" containerID="9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566" Oct 30 15:59:05.945603 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:05.945568 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566\": container with ID starting with 9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566 not found: ID does not exist" containerID="9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566" Oct 30 15:59:05.945708 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:05.945614 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566"} err="failed to get container status \"9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566\": rpc error: code = NotFound desc = could not find container \"9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566\": container with ID starting with 9995ed6b1406ec1e30b090b1d60627a3a3ca021da91de779a2c79b38e3a1b566 not found: ID does not exist" Oct 30 15:59:06.004846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.004786 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4"] Oct 30 15:59:06.010545 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.010507 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-2-w2-9e2b29-kserve-5f996b8df9-tm8r4"] Oct 30 15:59:06.048753 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.048708 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985"] Oct 30 15:59:06.058067 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.058033 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/simulator-w2-9e2b29-kserve-bbd645884-bf985"] Oct 30 15:59:06.098057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.098020 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp"] Oct 30 15:59:06.104388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.104350 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-9e2b29/distinct-w2-9e2b29-kserve-b66bb88-48xhp"] Oct 30 15:59:06.140587 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.140551 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf"] Oct 30 15:59:06.148692 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.148656 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/simulator-w1-a2b3f1-kserve-6974f9dbc-b5kxf"] Oct 30 15:59:06.192072 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.192034 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh"] Oct 30 15:59:06.201412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.201375 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a2b3f1/distinct-2-w1-a2b3f1-kserve-5cc57b6d46-hj2lh"] Oct 30 15:59:06.848367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.848330 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:59:06.848774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.848721 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:59:06.849553 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:06.849534 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 15:59:07.429226 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:07.429192 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" path="/var/lib/kubelet/pods/6dcf71f9-764f-4251-bf81-bb3222b85808/volumes" Oct 30 15:59:07.429655 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:07.429641 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" path="/var/lib/kubelet/pods/a129bf09-98a5-49c7-a89a-a4fda9f17ee3/volumes" Oct 30 15:59:07.430065 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:07.430050 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" path="/var/lib/kubelet/pods/a2f5c3db-a820-403d-a478-91c0ebe886c0/volumes" Oct 30 15:59:07.430748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:07.430731 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" path="/var/lib/kubelet/pods/a87bdf18-4e99-49df-aac9-1d499413de55/volumes" Oct 30 15:59:07.431173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:07.431158 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" path="/var/lib/kubelet/pods/b1f10fad-1159-4cc9-88b7-9da9159371a1/volumes" Oct 30 15:59:11.127493 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.127449 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" podUID="66b3a931-864d-44fe-99d6-813f944c8a3b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:11.891801 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.891759 4401 scope.go:122] "RemoveContainer" containerID="d18d86d4ad4b6e71156fe0eff33976b3f1d70d27d511453bc713b12dd2c0bb8c" Oct 30 15:59:11.903057 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.903034 4401 scope.go:122] "RemoveContainer" containerID="1dcdc1c7eb301d3e59c00bd809a9be9a5e9e4c73a07d94860348ab9a6762986a" Oct 30 15:59:11.913910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.913883 4401 scope.go:122] "RemoveContainer" containerID="c591e1955a4c757786653eb9334abc4b101c79a552ce00a4dd7c2aa3fcaba67e" Oct 30 15:59:11.924724 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.924699 4401 scope.go:122] "RemoveContainer" containerID="92ef94173977acda1e98d200a3f2ad82ab98916cd2bceae0983e7d76d6efb074" Oct 30 15:59:11.935470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.935446 4401 scope.go:122] "RemoveContainer" containerID="ffe6fac3124b55a6c445a298f707145fbf19bd80ede82a98e20a10e605f017aa" Oct 30 15:59:11.945978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.945943 4401 scope.go:122] "RemoveContainer" containerID="05ebbe6fc3935be6e774ac9996d7f6ca7bc8b7b128205c398e6a4427f6e0e147" Oct 30 15:59:11.956674 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.956644 4401 scope.go:122] "RemoveContainer" containerID="f7711b3d6ec882568db0b77a6e1fd69869c9d93a70972d1a90c90652dfe9d83a" Oct 30 15:59:11.967315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:11.967289 4401 scope.go:122] "RemoveContainer" containerID="16d2e0026ca55577a41e90e5e0b961eca2ac92d01b6f9a74dffaf618aba12425" Oct 30 15:59:12.134647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.134615 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:12.135027 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.134657 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" podUID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:12.566445 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.566408 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m"] Oct 30 15:59:12.567054 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567038 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" containerName="revoke-keys" Oct 30 15:59:12.567054 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567055 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" containerName="revoke-keys" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567079 4401 cpu_manager.go:414] "containerMap: removing container" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567084 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567102 4401 cpu_manager.go:414] "containerMap: removing container" podUID="9da206b1-06a1-44fa-9a61-29211b75cc57" containerName="revoke-keys" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567108 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da206b1-06a1-44fa-9a61-29211b75cc57" containerName="revoke-keys" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567115 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567120 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567132 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567136 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567146 4401 cpu_manager.go:414] "containerMap: removing container" podUID="f31ffe21-4522-4e15-bb9e-e9e0bfe13225" containerName="revoke-keys" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567150 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f31ffe21-4522-4e15-bb9e-e9e0bfe13225" containerName="revoke-keys" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567160 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567164 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" Oct 30 15:59:12.567179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567181 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" containerName="main" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567186 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" containerName="main" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567266 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dcf71f9-764f-4251-bf81-bb3222b85808" containerName="main" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567276 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da206b1-06a1-44fa-9a61-29211b75cc57" containerName="revoke-keys" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567283 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f10fad-1159-4cc9-88b7-9da9159371a1" containerName="main" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567290 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a129bf09-98a5-49c7-a89a-a4fda9f17ee3" containerName="main" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567297 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="f31ffe21-4522-4e15-bb9e-e9e0bfe13225" containerName="revoke-keys" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567304 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2f5c3db-a820-403d-a478-91c0ebe886c0" containerName="main" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567311 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a87bdf18-4e99-49df-aac9-1d499413de55" containerName="main" Oct 30 15:59:12.567557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.567317 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" containerName="revoke-keys" Oct 30 15:59:12.575050 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.575022 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.597985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.597952 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m"] Oct 30 15:59:12.619713 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.619677 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.619924 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.619816 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.619970 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.619919 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.620022 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.620003 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.620055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.620034 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8jn\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-kube-api-access-rt8jn\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.620091 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.620080 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f8600c-0b67-455c-aae8-d506e529300b-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.620206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.620194 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.620258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.620246 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.620290 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.620271 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f8600c-0b67-455c-aae8-d506e529300b-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.721998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.721957 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.721998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722003 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722043 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722060 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8jn\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-kube-api-access-rt8jn\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722099 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f8600c-0b67-455c-aae8-d506e529300b-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722157 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722231 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722264 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f8600c-0b67-455c-aae8-d506e529300b-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722588 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722306 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722588 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722426 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722588 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722471 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722588 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722527 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722744 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722608 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.722877 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.722819 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f8600c-0b67-455c-aae8-d506e529300b-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.724599 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.724570 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.724809 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.724788 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f8600c-0b67-455c-aae8-d506e529300b-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.762814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.762777 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.762814 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.762804 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8jn\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-kube-api-access-rt8jn\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:12.901916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:12.901877 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:13.141302 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:13.141264 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" podUID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:59:13.231751 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:59:13.231722 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f8600c_0b67_455c_aae8_d506e529300b.slice/crio-fffb2d66d2901353528aa3af61cd67bb3b304d86634488dc3eeb7760da863e06 WatchSource:0}: Error finding container fffb2d66d2901353528aa3af61cd67bb3b304d86634488dc3eeb7760da863e06: Status 404 returned error can't find the container with id fffb2d66d2901353528aa3af61cd67bb3b304d86634488dc3eeb7760da863e06 Oct 30 15:59:13.233604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:13.233576 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m"] Oct 30 15:59:13.234104 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:13.234070 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:59:13.234186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:13.234136 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:59:13.234186 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:13.234169 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:59:13.814602 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:13.814547 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:59:14.020610 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:14.020571 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" event={"ID":"27f8600c-0b67-455c-aae8-d506e529300b","Type":"ContainerStarted","Data":"c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0"} Oct 30 15:59:14.020817 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:14.020630 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" event={"ID":"27f8600c-0b67-455c-aae8-d506e529300b","Type":"ContainerStarted","Data":"fffb2d66d2901353528aa3af61cd67bb3b304d86634488dc3eeb7760da863e06"} Oct 30 15:59:14.095070 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:14.094931 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podStartSLOduration=2.094909862 podStartE2EDuration="2.094909862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:59:14.087235867 +0000 UTC m=+1262.786509025" watchObservedRunningTime="2026-10-30 15:59:14.094909862 +0000 UTC m=+1262.794183019" Oct 30 15:59:14.904999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:14.904953 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:14.905554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:14.905031 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:14.905554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:14.905436 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:15.904443 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:15.904403 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:15.904749 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:15.904713 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:16.903458 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:16.903417 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:16.904100 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:16.904067 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:17.907853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:17.904272 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:17.907853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:17.904352 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:17.959910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:17.936424 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb"] Oct 30 15:59:18.002147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:17.986668 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-040f9a-bk9qb"] Oct 30 15:59:18.811866 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:18.794628 4401 generic.go:356] "Generic (PLEG): container finished" podID="66b3a931-864d-44fe-99d6-813f944c8a3b" containerID="b82affa4b62a9bb0042f60b084677bde3228817daa1dd419e402e28f386fa4d4" exitCode=0 Oct 30 15:59:18.812426 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:18.806991 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" event={"ID":"66b3a931-864d-44fe-99d6-813f944c8a3b","Type":"ContainerDied","Data":"b82affa4b62a9bb0042f60b084677bde3228817daa1dd419e402e28f386fa4d4"} Oct 30 15:59:18.904865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:18.904283 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:18.904865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:18.904384 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:19.040173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:19.034880 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" event={"ID":"5735b6bb-a75e-4aee-9d7d-7cf796abe26c","Type":"ContainerDied","Data":"e1a96dc874ec5970db728e073417a84c32bf791d371552af821715939ade6932"} Oct 30 15:59:19.062554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:19.021062 4401 generic.go:356] "Generic (PLEG): container finished" podID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" containerID="e1a96dc874ec5970db728e073417a84c32bf791d371552af821715939ade6932" exitCode=0 Oct 30 15:59:19.561800 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:19.560701 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:59:19.988959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:19.973782 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37" path="/var/lib/kubelet/pods/dc30ecf6-f9d2-45af-aaaf-a5d63e52fd37/volumes" Oct 30 15:59:19.988959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:19.974304 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm"] Oct 30 15:59:20.012617 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.012573 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b3a931-864d-44fe-99d6-813f944c8a3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b3a931-864d-44fe-99d6-813f944c8a3b-plugins-config-volume\") pod \"66b3a931-864d-44fe-99d6-813f944c8a3b\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " Oct 30 15:59:20.013095 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.013067 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b3a931-864d-44fe-99d6-813f944c8a3b-kube-api-access-4txkh\" (UniqueName: \"kubernetes.io/projected/66b3a931-864d-44fe-99d6-813f944c8a3b-kube-api-access-4txkh\") pod \"66b3a931-864d-44fe-99d6-813f944c8a3b\" (UID: \"66b3a931-864d-44fe-99d6-813f944c8a3b\") " Oct 30 15:59:20.060878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.051661 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b3a931-864d-44fe-99d6-813f944c8a3b-plugins-config-volume" pod "66b3a931-864d-44fe-99d6-813f944c8a3b" (UID: "66b3a931-864d-44fe-99d6-813f944c8a3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:20.060878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.051764 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:20.060878 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.051817 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:20.179926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.159663 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-9e2b29-gd8xm"] Oct 30 15:59:20.179926 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.172177 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b3a931-864d-44fe-99d6-813f944c8a3b-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:20.344050 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.343992 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b3a931-864d-44fe-99d6-813f944c8a3b-kube-api-access-4txkh" pod "66b3a931-864d-44fe-99d6-813f944c8a3b" (UID: "66b3a931-864d-44fe-99d6-813f944c8a3b"). InnerVolumeSpecName "kube-api-access-4txkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:20.360668 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.360631 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4txkh\" (UniqueName: \"kubernetes.io/projected/66b3a931-864d-44fe-99d6-813f944c8a3b-kube-api-access-4txkh\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:20.380432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.380398 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:59:20.446976 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.437483 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" event={"ID":"66b3a931-864d-44fe-99d6-813f944c8a3b","Type":"ContainerDied","Data":"b73157fb086ed5a4aa9dc4db0de04c767751de8bfce81221544dfcbedc81b6ff"} Oct 30 15:59:20.446976 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.437562 4401 scope.go:122] "RemoveContainer" containerID="b82affa4b62a9bb0042f60b084677bde3228817daa1dd419e402e28f386fa4d4" Oct 30 15:59:20.446976 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.437751 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q" Oct 30 15:59:20.471968 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.471924 4401 generic.go:356] "Generic (PLEG): container finished" podID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" containerID="e113cb88be99df19ad677cb27ee659d7e72977caae97ff990d3627db134e729e" exitCode=0 Oct 30 15:59:20.476547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.476484 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" event={"ID":"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b","Type":"ContainerDied","Data":"e113cb88be99df19ad677cb27ee659d7e72977caae97ff990d3627db134e729e"} Oct 30 15:59:20.513647 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.513612 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-kube-api-access-mlbp2\" (UniqueName: \"kubernetes.io/projected/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-kube-api-access-mlbp2\") pod \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " Oct 30 15:59:20.513997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.513974 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-plugins-config-volume\") pod \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\" (UID: \"5735b6bb-a75e-4aee-9d7d-7cf796abe26c\") " Oct 30 15:59:20.539976 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.538333 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:59:20.539976 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.538394 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:59:20.545254 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.545202 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-plugins-config-volume" pod "5735b6bb-a75e-4aee-9d7d-7cf796abe26c" (UID: "5735b6bb-a75e-4aee-9d7d-7cf796abe26c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:20.681171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.681116 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:20.833850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.818786 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" Oct 30 15:59:20.833850 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.819330 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j" event={"ID":"5735b6bb-a75e-4aee-9d7d-7cf796abe26c","Type":"ContainerDied","Data":"8ded4521a521e5c166cb12f222c9698d6603eee80511fd249846968619566f8b"} Oct 30 15:59:20.868273 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.856881 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q"] Oct 30 15:59:20.868273 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.858926 4401 scope.go:122] "RemoveContainer" containerID="e1a96dc874ec5970db728e073417a84c32bf791d371552af821715939ade6932" Oct 30 15:59:20.920851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.918179 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:20.920851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.918250 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:20.979440 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:20.979285 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-kube-api-access-mlbp2" pod "5735b6bb-a75e-4aee-9d7d-7cf796abe26c" (UID: "5735b6bb-a75e-4aee-9d7d-7cf796abe26c"). InnerVolumeSpecName "kube-api-access-mlbp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:21.009922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.009748 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-040f9a-bfb64fb4-bng8q"] Oct 30 15:59:21.041112 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.033956 4401 patch_prober.go:29] interesting pod/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:59:21.041112 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.034036 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:59:21.061851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.058889 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbp2\" (UniqueName: \"kubernetes.io/projected/5735b6bb-a75e-4aee-9d7d-7cf796abe26c-kube-api-access-mlbp2\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:21.301922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.297959 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:59:21.301922 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.298034 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:59:21.627354 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.627311 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b3a931-864d-44fe-99d6-813f944c8a3b" path="/var/lib/kubelet/pods/66b3a931-864d-44fe-99d6-813f944c8a3b/volumes" Oct 30 15:59:21.628313 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.628285 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faadbed6-ccc3-4b9b-8812-2aa80cd9eb00" path="/var/lib/kubelet/pods/faadbed6-ccc3-4b9b-8812-2aa80cd9eb00/volumes" Oct 30 15:59:21.907500 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.907401 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:21.907776 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.907740 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:21.931616 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:21.931580 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:59:22.008194 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.008147 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j"] Oct 30 15:59:22.012893 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.012852 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-040f9a-d4fb5b7ff-rpl8j"] Oct 30 15:59:22.015089 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.015048 4401 generic.go:356] "Generic (PLEG): container finished" podID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerID="7a07aa5051c09b22ff7780ef870534aa1cf317acd5db1c1df64cca1680adf417" exitCode=0 Oct 30 15:59:22.015275 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.015166 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" event={"ID":"4788e294-9ac9-462f-bcad-5fdfdafd2d20","Type":"ContainerDied","Data":"7a07aa5051c09b22ff7780ef870534aa1cf317acd5db1c1df64cca1680adf417"} Oct 30 15:59:22.030849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.028221 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" event={"ID":"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b","Type":"ContainerDied","Data":"14418f5550b213d11bd606095e92dc3a79ed3befcf57740a7211854e44abecad"} Oct 30 15:59:22.030849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.028360 4401 scope.go:122] "RemoveContainer" containerID="e113cb88be99df19ad677cb27ee659d7e72977caae97ff990d3627db134e729e" Oct 30 15:59:22.030849 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.028553 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56" Oct 30 15:59:22.057726 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.057682 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-plugins-config-volume\") pod \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " Oct 30 15:59:22.058302 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.058270 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-kube-api-access-kcdc5\" (UniqueName: \"kubernetes.io/projected/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-kube-api-access-kcdc5\") pod \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\" (UID: \"faa18168-1e2a-4b93-8a6a-36ab2bcbab7b\") " Oct 30 15:59:22.066843 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:22.066742 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:59:22.078225 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.074261 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-plugins-config-volume" pod "faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" (UID: "faa18168-1e2a-4b93-8a6a-36ab2bcbab7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:22.115774 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.114261 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-kube-api-access-kcdc5" pod "faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" (UID: "faa18168-1e2a-4b93-8a6a-36ab2bcbab7b"). InnerVolumeSpecName "kube-api-access-kcdc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:22.183036 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.182938 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcdc5\" (UniqueName: \"kubernetes.io/projected/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-kube-api-access-kcdc5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:22.183036 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.182978 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:22.197668 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.197637 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:59:22.284389 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.284353 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4788e294-9ac9-462f-bcad-5fdfdafd2d20-kube-api-access-hppxj\" (UniqueName: \"kubernetes.io/projected/4788e294-9ac9-462f-bcad-5fdfdafd2d20-kube-api-access-hppxj\") pod \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " Oct 30 15:59:22.284604 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.284486 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4788e294-9ac9-462f-bcad-5fdfdafd2d20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4788e294-9ac9-462f-bcad-5fdfdafd2d20-plugins-config-volume\") pod \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\" (UID: \"4788e294-9ac9-462f-bcad-5fdfdafd2d20\") " Oct 30 15:59:22.285029 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.284997 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4788e294-9ac9-462f-bcad-5fdfdafd2d20-plugins-config-volume" pod "4788e294-9ac9-462f-bcad-5fdfdafd2d20" (UID: "4788e294-9ac9-462f-bcad-5fdfdafd2d20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:22.286626 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.286594 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4788e294-9ac9-462f-bcad-5fdfdafd2d20-kube-api-access-hppxj" pod "4788e294-9ac9-462f-bcad-5fdfdafd2d20" (UID: "4788e294-9ac9-462f-bcad-5fdfdafd2d20"). InnerVolumeSpecName "kube-api-access-hppxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:22.386081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.386045 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4788e294-9ac9-462f-bcad-5fdfdafd2d20-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:22.386081 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.386077 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hppxj\" (UniqueName: \"kubernetes.io/projected/4788e294-9ac9-462f-bcad-5fdfdafd2d20-kube-api-access-hppxj\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:22.393729 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.393693 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56"] Oct 30 15:59:22.399154 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.399120 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a2b3f1-f4d459f46-ddb56"] Oct 30 15:59:22.903052 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.903010 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:22.903242 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:22.903091 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:23.090949 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.090906 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" event={"ID":"4788e294-9ac9-462f-bcad-5fdfdafd2d20","Type":"ContainerDied","Data":"65eaa254afce31bec701880a526417213f70d4a43bd2254fdf821b7745b9ed44"} Oct 30 15:59:23.094997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.094966 4401 scope.go:122] "RemoveContainer" containerID="7a07aa5051c09b22ff7780ef870534aa1cf317acd5db1c1df64cca1680adf417" Oct 30 15:59:23.095216 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.091761 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:59:23.136312 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.136272 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.61:50052\" within 1s: context deadline exceeded" Oct 30 15:59:23.136478 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.136358 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp" Oct 30 15:59:23.136478 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:59:23.136393 4401 logging.go:55] [core] [Channel #1784 SubChannel #1785]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.61:50052", ServerName: "10.133.0.61:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.61:50052: operation was canceled" Oct 30 15:59:23.181912 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.180930 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp"] Oct 30 15:59:23.243856 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.240485 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-9e2b29-74b8dd9b8-gjgkp"] Oct 30 15:59:23.434213 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.434123 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" path="/var/lib/kubelet/pods/4788e294-9ac9-462f-bcad-5fdfdafd2d20/volumes" Oct 30 15:59:23.434834 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.434802 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" path="/var/lib/kubelet/pods/5735b6bb-a75e-4aee-9d7d-7cf796abe26c/volumes" Oct 30 15:59:23.435386 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.435365 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" path="/var/lib/kubelet/pods/faa18168-1e2a-4b93-8a6a-36ab2bcbab7b/volumes" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.705690 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc"] Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708465 4401 cpu_manager.go:414] "containerMap: removing container" podUID="66b3a931-864d-44fe-99d6-813f944c8a3b" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708498 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b3a931-864d-44fe-99d6-813f944c8a3b" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708513 4401 cpu_manager.go:414] "containerMap: removing container" podUID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708521 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708567 4401 cpu_manager.go:414] "containerMap: removing container" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708576 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708595 4401 cpu_manager.go:414] "containerMap: removing container" podUID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" containerName="payload-pre-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708603 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" containerName="payload-pre-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708798 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b3a931-864d-44fe-99d6-813f944c8a3b" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708813 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa18168-1e2a-4b93-8a6a-36ab2bcbab7b" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708846 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="4788e294-9ac9-462f-bcad-5fdfdafd2d20" containerName="payload-processing" Oct 30 15:59:23.712954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.708860 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="5735b6bb-a75e-4aee-9d7d-7cf796abe26c" containerName="payload-pre-processing" Oct 30 15:59:23.738478 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.738438 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc"] Oct 30 15:59:23.753846 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.738839 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.785660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.785151 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:59:23.785660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.785406 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:59:23.785660 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.785568 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:59:23.793261 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.786415 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:59:23.830875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.830667 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8kp\" (UniqueName: \"kubernetes.io/projected/e5420580-a0a7-4211-be18-7c45a354508d-kube-api-access-mk8kp\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.830875 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.830862 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.831137 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.830956 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.899706 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.899672 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj"] Oct 30 15:59:23.899950 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.899899 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" containerID="cri-o://a6b23b5dd1ee838c037e2c8766a87800e755a8149d1a2c10eda9fb4574b2242d" gracePeriod=30 Oct 30 15:59:23.902863 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.902784 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:23.903020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.902888 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:23.906357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.906317 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn"] Oct 30 15:59:23.906593 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.906564 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" containerID="cri-o://305b5530e0c66efa37bfcacecd9b511c3702080d5fa63639065d0c13012644c9" gracePeriod=30 Oct 30 15:59:23.912411 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.912376 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26"] Oct 30 15:59:23.912624 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.912600 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" containerID="cri-o://1308ddb569752d302b4db1c4b5bc92a11b3cf65fcca61a903fa92df357f4b59e" gracePeriod=30 Oct 30 15:59:23.919179 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.919144 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f"] Oct 30 15:59:23.919418 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.919393 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" containerID="cri-o://0720ca1dc0f92df33ca0e213ccc4cf14494723dc0c64f22450c49fd583fe8ad4" gracePeriod=30 Oct 30 15:59:23.932464 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.932430 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8kp\" (UniqueName: \"kubernetes.io/projected/e5420580-a0a7-4211-be18-7c45a354508d-kube-api-access-mk8kp\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.932627 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.932570 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.932678 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.932664 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.933381 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.933355 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.935013 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.934990 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:23.970700 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:23.970621 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8kp\" (UniqueName: \"kubernetes.io/projected/e5420580-a0a7-4211-be18-7c45a354508d-kube-api-access-mk8kp\") pod \"maas-api-revoke-sub-8325bc7d-l88sc\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:24.144438 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:24.144399 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:24.459625 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:59:24.459585 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5420580_a0a7_4211_be18_7c45a354508d.slice/crio-dd8e31d5cae45c5b5f5f565b4e0dae145a504bf717229401ab9ab4bd5675fb4a WatchSource:0}: Error finding container dd8e31d5cae45c5b5f5f565b4e0dae145a504bf717229401ab9ab4bd5675fb4a: Status 404 returned error can't find the container with id dd8e31d5cae45c5b5f5f565b4e0dae145a504bf717229401ab9ab4bd5675fb4a Oct 30 15:59:24.460059 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:24.460034 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc"] Oct 30 15:59:24.903009 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:24.902970 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body= Oct 30 15:59:24.903220 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:24.903034 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" Oct 30 15:59:25.134336 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:25.134289 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" event={"ID":"e5420580-a0a7-4211-be18-7c45a354508d","Type":"ContainerStarted","Data":"6bbad3f9ac183960acc1240af9659fc68ae1f86b267ef54590db85d7dea54358"} Oct 30 15:59:25.134552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:25.134399 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" event={"ID":"e5420580-a0a7-4211-be18-7c45a354508d","Type":"ContainerStarted","Data":"dd8e31d5cae45c5b5f5f565b4e0dae145a504bf717229401ab9ab4bd5675fb4a"} Oct 30 15:59:25.179693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:25.179631 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" podStartSLOduration=2.17961725 podStartE2EDuration="2.17961725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:59:25.178643922 +0000 UTC m=+1273.877917080" watchObservedRunningTime="2026-10-30 15:59:25.17961725 +0000 UTC m=+1273.878890406" Oct 30 15:59:26.145026 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.144983 4401 generic.go:356] "Generic (PLEG): container finished" podID="e5420580-a0a7-4211-be18-7c45a354508d" containerID="6bbad3f9ac183960acc1240af9659fc68ae1f86b267ef54590db85d7dea54358" exitCode=0 Oct 30 15:59:26.145248 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.145083 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" event={"ID":"e5420580-a0a7-4211-be18-7c45a354508d","Type":"ContainerDied","Data":"6bbad3f9ac183960acc1240af9659fc68ae1f86b267ef54590db85d7dea54358"} Oct 30 15:59:26.689793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.688864 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj"] Oct 30 15:59:26.689793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.689132 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" containerID="cri-o://48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49" gracePeriod=30 Oct 30 15:59:26.704188 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.703204 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm"] Oct 30 15:59:26.704188 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.703444 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" containerID="cri-o://ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3" gracePeriod=30 Oct 30 15:59:26.713507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.713456 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd"] Oct 30 15:59:26.714083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.714009 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" containerID="cri-o://637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd" gracePeriod=30 Oct 30 15:59:26.953935 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.952941 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:59:26.953935 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:26.953014 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:59:27.489326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.488400 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7"] Oct 30 15:59:27.489326 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.488653 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" containerID="cri-o://1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327" gracePeriod=30 Oct 30 15:59:27.522537 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.522432 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:27.527965 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.527934 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm"] Oct 30 15:59:27.529609 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.529583 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:27.530460 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.530044 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" containerID="cri-o://00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491" gracePeriod=30 Oct 30 15:59:27.533020 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.532988 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 15:59:27.573416 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.573382 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:27.670931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.667336 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5420580-a0a7-4211-be18-7c45a354508d-kube-api-access-mk8kp\" (UniqueName: \"kubernetes.io/projected/e5420580-a0a7-4211-be18-7c45a354508d-kube-api-access-mk8kp\") pod \"e5420580-a0a7-4211-be18-7c45a354508d\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " Oct 30 15:59:27.670931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.667508 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-openshift-ca\") pod \"e5420580-a0a7-4211-be18-7c45a354508d\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " Oct 30 15:59:27.670931 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.667590 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-xks-ca\") pod \"e5420580-a0a7-4211-be18-7c45a354508d\" (UID: \"e5420580-a0a7-4211-be18-7c45a354508d\") " Oct 30 15:59:27.705474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.705098 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5420580-a0a7-4211-be18-7c45a354508d-kube-api-access-mk8kp" pod "e5420580-a0a7-4211-be18-7c45a354508d" (UID: "e5420580-a0a7-4211-be18-7c45a354508d"). InnerVolumeSpecName "kube-api-access-mk8kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:27.731853 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.708313 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-xks-ca" pod "e5420580-a0a7-4211-be18-7c45a354508d" (UID: "e5420580-a0a7-4211-be18-7c45a354508d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:27.756287 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.755554 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd"] Oct 30 15:59:27.756881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.756521 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e5420580-a0a7-4211-be18-7c45a354508d" containerName="revoke-keys" Oct 30 15:59:27.756881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.756549 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5420580-a0a7-4211-be18-7c45a354508d" containerName="revoke-keys" Oct 30 15:59:27.756881 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.756806 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5420580-a0a7-4211-be18-7c45a354508d" containerName="revoke-keys" Oct 30 15:59:27.759281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.756986 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:27.759281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.757012 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8kp\" (UniqueName: \"kubernetes.io/projected/e5420580-a0a7-4211-be18-7c45a354508d-kube-api-access-mk8kp\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:27.759281 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.757508 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-openshift-ca" pod "e5420580-a0a7-4211-be18-7c45a354508d" (UID: "e5420580-a0a7-4211-be18-7c45a354508d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:27.774130 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.774054 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd"] Oct 30 15:59:27.774268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.774168 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.857989 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.857936 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858204 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858017 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858204 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858058 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2j5\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-kube-api-access-qr2j5\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858204 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858087 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858202 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858357 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858306 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858434 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858367 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858483 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858467 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858536 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858497 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.858742 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.858723 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5420580-a0a7-4211-be18-7c45a354508d-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:27.959567 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959529 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2j5\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-kube-api-access-qr2j5\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.959793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959584 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.959793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959637 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.959793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959691 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.959793 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959736 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959816 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959864 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.959979 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960085 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.960047 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960308 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.960173 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960465 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.960436 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-credential-socket\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960552 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.960531 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-data\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.960611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.960582 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-certs\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.961029 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.961005 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istiod-ca-cert\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.962378 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.962344 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-envoy\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.962584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.962560 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-podinfo\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.985554 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.985515 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2j5\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-kube-api-access-qr2j5\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:27.986034 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:27.986012 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-token\") pod \"e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:28.124848 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.123304 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:28.194961 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.194930 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" Oct 30 15:59:28.195630 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.195600 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc" event={"ID":"e5420580-a0a7-4211-be18-7c45a354508d","Type":"ContainerDied","Data":"dd8e31d5cae45c5b5f5f565b4e0dae145a504bf717229401ab9ab4bd5675fb4a"} Oct 30 15:59:28.195892 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.195873 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd8e31d5cae45c5b5f5f565b4e0dae145a504bf717229401ab9ab4bd5675fb4a" Oct 30 15:59:28.503563 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:59:28.503530 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3df45a1_fa98_4c66_82a1_f17e2d1f42f3.slice/crio-e8cea90f35bb2dc93f35ea841aa53dfe3acd8196d6e2c3a80346b8daad7dace7 WatchSource:0}: Error finding container e8cea90f35bb2dc93f35ea841aa53dfe3acd8196d6e2c3a80346b8daad7dace7: Status 404 returned error can't find the container with id e8cea90f35bb2dc93f35ea841aa53dfe3acd8196d6e2c3a80346b8daad7dace7 Oct 30 15:59:28.505160 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.505134 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd"] Oct 30 15:59:28.506288 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.506249 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:59:28.506364 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.506339 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:59:28.506412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:28.506375 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 15:59:29.212546 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.212500 4401 generic.go:356] "Generic (PLEG): container finished" podID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerID="1308ddb569752d302b4db1c4b5bc92a11b3cf65fcca61a903fa92df357f4b59e" exitCode=0 Oct 30 15:59:29.213330 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.212679 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" event={"ID":"29c5b157-a5fd-4560-b25f-25719b24fef7","Type":"ContainerDied","Data":"1308ddb569752d302b4db1c4b5bc92a11b3cf65fcca61a903fa92df357f4b59e"} Oct 30 15:59:29.217030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.216940 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" event={"ID":"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3","Type":"ContainerStarted","Data":"ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673"} Oct 30 15:59:29.217030 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.217010 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" event={"ID":"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3","Type":"ContainerStarted","Data":"e8cea90f35bb2dc93f35ea841aa53dfe3acd8196d6e2c3a80346b8daad7dace7"} Oct 30 15:59:29.222714 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.222676 4401 generic.go:356] "Generic (PLEG): container finished" podID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerID="305b5530e0c66efa37bfcacecd9b511c3702080d5fa63639065d0c13012644c9" exitCode=0 Oct 30 15:59:29.222919 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.222816 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" event={"ID":"6cdc8085-eee2-4c42-9d59-8935888b5d65","Type":"ContainerDied","Data":"305b5530e0c66efa37bfcacecd9b511c3702080d5fa63639065d0c13012644c9"} Oct 30 15:59:29.225946 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.225912 4401 generic.go:356] "Generic (PLEG): container finished" podID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerID="0720ca1dc0f92df33ca0e213ccc4cf14494723dc0c64f22450c49fd583fe8ad4" exitCode=0 Oct 30 15:59:29.226103 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.226045 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" event={"ID":"b364f9b9-6f62-4bd2-b1f8-9636e024529a","Type":"ContainerDied","Data":"0720ca1dc0f92df33ca0e213ccc4cf14494723dc0c64f22450c49fd583fe8ad4"} Oct 30 15:59:29.226103 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.226099 4401 scope.go:122] "RemoveContainer" containerID="d741463923f2d50316e10d53e1f9192b6770b381cc2336d84508014b82070d89" Oct 30 15:59:29.238295 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.237209 4401 generic.go:356] "Generic (PLEG): container finished" podID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerID="a6b23b5dd1ee838c037e2c8766a87800e755a8149d1a2c10eda9fb4574b2242d" exitCode=0 Oct 30 15:59:29.239123 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.239028 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" event={"ID":"80a4faf0-b580-4e3b-b71c-cda031c990f9","Type":"ContainerDied","Data":"a6b23b5dd1ee838c037e2c8766a87800e755a8149d1a2c10eda9fb4574b2242d"} Oct 30 15:59:29.294999 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.294969 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:59:29.301983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.301953 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:59:29.310169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.310111 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podStartSLOduration=2.310089633 podStartE2EDuration="2.310089633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:59:29.292918559 +0000 UTC m=+1277.992191716" watchObservedRunningTime="2026-10-30 15:59:29.310089633 +0000 UTC m=+1278.009362792" Oct 30 15:59:29.384938 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.384877 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-kube-api-access-8mmrf\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-kube-api-access-8mmrf\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.386971 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-podinfo\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387070 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-certs\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387103 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-token\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-token\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387132 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-token\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-token\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387175 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-data\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387243 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-data\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387328 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cdc8085-eee2-4c42-9d59-8935888b5d65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc8085-eee2-4c42-9d59-8935888b5d65-istiod-ca-cert\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387397 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-envoy\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387429 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-kube-api-access-6jkgv\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-kube-api-access-6jkgv\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387475 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-certs\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387556 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a4faf0-b580-4e3b-b71c-cda031c990f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a4faf0-b580-4e3b-b71c-cda031c990f9-istiod-ca-cert\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387600 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-podinfo\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387666 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-envoy\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387709 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-socket\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387770 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-socket\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387879 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-credential-socket\") pod \"80a4faf0-b580-4e3b-b71c-cda031c990f9\" (UID: \"80a4faf0-b580-4e3b-b71c-cda031c990f9\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.387927 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-credential-socket\") pod \"6cdc8085-eee2-4c42-9d59-8935888b5d65\" (UID: \"6cdc8085-eee2-4c42-9d59-8935888b5d65\") " Oct 30 15:59:29.391021 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.389088 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-credential-socket" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.392333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.391768 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-certs" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.392333 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.391900 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-kube-api-access-8mmrf" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "kube-api-access-8mmrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.392589 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.392558 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-data" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.392985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.392805 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-certs" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.392985 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.392871 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a4faf0-b580-4e3b-b71c-cda031c990f9-istiod-ca-cert" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:29.395105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.393679 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-socket" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.395105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.393788 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-data" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.395105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.394864 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cdc8085-eee2-4c42-9d59-8935888b5d65-istiod-ca-cert" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:29.395105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.394992 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-socket" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.395105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.395049 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-credential-socket" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.395631 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.395599 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-podinfo" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:29.395928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.395907 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-envoy" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.396463 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.396438 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-podinfo" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:29.396581 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.396555 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-token" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.396658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.396595 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-token" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.396698 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.396665 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-envoy" pod "80a4faf0-b580-4e3b-b71c-cda031c990f9" (UID: "80a4faf0-b580-4e3b-b71c-cda031c990f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.397858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.397814 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-kube-api-access-6jkgv" pod "6cdc8085-eee2-4c42-9d59-8935888b5d65" (UID: "6cdc8085-eee2-4c42-9d59-8935888b5d65"). InnerVolumeSpecName "kube-api-access-6jkgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.436954 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.436886 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:59:29.442083 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.441782 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:59:29.489910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.489866 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-kube-api-access-f64tt\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-kube-api-access-f64tt\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.489910 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.489912 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-podinfo\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.489937 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-socket\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.489966 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-kube-api-access-hbbq5\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-kube-api-access-hbbq5\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490076 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-token\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-token\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490117 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-certs\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490163 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490154 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-envoy\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490180 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-envoy\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490213 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-socket\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490232 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-socket" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.490422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490271 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-certs\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490326 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-credential-socket\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490364 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istiod-ca-cert\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490422 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490404 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-podinfo\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490427 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-certs" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490487 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-token\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-token\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490523 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-data\") pod \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\" (UID: \"b364f9b9-6f62-4bd2-b1f8-9636e024529a\") " Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490549 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-certs" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490556 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-data\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490674 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c5b157-a5fd-4560-b25f-25719b24fef7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c5b157-a5fd-4560-b25f-25719b24fef7-istiod-ca-cert\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490741 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-credential-socket\") pod \"29c5b157-a5fd-4560-b25f-25719b24fef7\" (UID: \"29c5b157-a5fd-4560-b25f-25719b24fef7\") " Oct 30 15:59:29.490792 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490764 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-data" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.491594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490951 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-socket" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.491594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.490967 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-credential-socket" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.491594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491162 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-credential-socket" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.491594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491237 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istiod-ca-cert" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:29.491594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491471 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-data" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.491594 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491561 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c5b157-a5fd-4560-b25f-25719b24fef7-istiod-ca-cert" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491765 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491787 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491801 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491816 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491866 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491891 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mmrf\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-kube-api-access-8mmrf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491910 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.491930 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491926 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491942 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491959 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491973 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491986 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.491999 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492011 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492025 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cdc8085-eee2-4c42-9d59-8935888b5d65-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492037 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492051 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492064 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jkgv\" (UniqueName: \"kubernetes.io/projected/6cdc8085-eee2-4c42-9d59-8935888b5d65-kube-api-access-6jkgv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492077 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492090 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492104 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492117 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a4faf0-b580-4e3b-b71c-cda031c990f9-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492130 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492145 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492161 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492175 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a4faf0-b580-4e3b-b71c-cda031c990f9-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.492348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.492188 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cdc8085-eee2-4c42-9d59-8935888b5d65-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.493299 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.493166 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-envoy" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.493299 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.493210 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-token" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.493395 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.493321 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-podinfo" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:29.493902 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.493878 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-envoy" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:29.493963 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.493902 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-kube-api-access-hbbq5" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "kube-api-access-hbbq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.494129 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.494109 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-kube-api-access-f64tt" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "kube-api-access-f64tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.494258 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.494244 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-token" pod "b364f9b9-6f62-4bd2-b1f8-9636e024529a" (UID: "b364f9b9-6f62-4bd2-b1f8-9636e024529a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:29.494408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.494393 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-podinfo" pod "29c5b157-a5fd-4560-b25f-25719b24fef7" (UID: "29c5b157-a5fd-4560-b25f-25719b24fef7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:29.593431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593395 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c5b157-a5fd-4560-b25f-25719b24fef7-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593426 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f64tt\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-kube-api-access-f64tt\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593431 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593439 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593451 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbbq5\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-kube-api-access-hbbq5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593463 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593474 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593485 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593496 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c5b157-a5fd-4560-b25f-25719b24fef7-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:29.593679 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:29.593507 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b364f9b9-6f62-4bd2-b1f8-9636e024529a-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:30.124584 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.124548 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:30.126105 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.126075 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:59:30.126250 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.126129 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 15:59:30.272573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.272538 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" Oct 30 15:59:30.272573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.272568 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn" event={"ID":"6cdc8085-eee2-4c42-9d59-8935888b5d65","Type":"ContainerDied","Data":"5c152c27f50d91bd47119f7f0b32b27f788ab44f05817c1bbfc98ce2866b815c"} Oct 30 15:59:30.273103 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.272632 4401 scope.go:122] "RemoveContainer" containerID="305b5530e0c66efa37bfcacecd9b511c3702080d5fa63639065d0c13012644c9" Oct 30 15:59:30.274891 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.274788 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" Oct 30 15:59:30.274891 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.274818 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f" event={"ID":"b364f9b9-6f62-4bd2-b1f8-9636e024529a","Type":"ContainerDied","Data":"9cec31c6f67754ece8629d692dcb115903b439bc94590a7b3c67e9e2059d6aa8"} Oct 30 15:59:30.277043 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.277020 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" Oct 30 15:59:30.277164 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.277018 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj" event={"ID":"80a4faf0-b580-4e3b-b71c-cda031c990f9","Type":"ContainerDied","Data":"c5076aa50392bcf2f37f0d2eb57fff86c1b39e9ccc2bac13061bc3d8182ceac0"} Oct 30 15:59:30.278928 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.278906 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" Oct 30 15:59:30.279054 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.278950 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26" event={"ID":"29c5b157-a5fd-4560-b25f-25719b24fef7","Type":"ContainerDied","Data":"2cc84536a1a2d61c78f12f7fad3cfe190a68c7e466769ccd0a36515d16003d02"} Oct 30 15:59:30.288711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.288688 4401 scope.go:122] "RemoveContainer" containerID="0720ca1dc0f92df33ca0e213ccc4cf14494723dc0c64f22450c49fd583fe8ad4" Oct 30 15:59:30.300715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.300689 4401 scope.go:122] "RemoveContainer" containerID="a6b23b5dd1ee838c037e2c8766a87800e755a8149d1a2c10eda9fb4574b2242d" Oct 30 15:59:30.311918 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.311894 4401 scope.go:122] "RemoveContainer" containerID="1308ddb569752d302b4db1c4b5bc92a11b3cf65fcca61a903fa92df357f4b59e" Oct 30 15:59:30.365173 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.365094 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f"] Oct 30 15:59:30.375577 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.375484 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-njd6f"] Oct 30 15:59:30.394916 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.394885 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn"] Oct 30 15:59:30.400622 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.400590 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-mrkpn"] Oct 30 15:59:30.476933 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.476891 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj"] Oct 30 15:59:30.482701 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.482672 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-fqjmj"] Oct 30 15:59:30.519501 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.519462 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26"] Oct 30 15:59:30.534044 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:30.525679 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-hcj26"] Oct 30 15:59:31.124864 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.124786 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:59:31.125056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.124892 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 15:59:31.429401 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.429319 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" path="/var/lib/kubelet/pods/29c5b157-a5fd-4560-b25f-25719b24fef7/volumes" Oct 30 15:59:31.429993 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.429972 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" path="/var/lib/kubelet/pods/6cdc8085-eee2-4c42-9d59-8935888b5d65/volumes" Oct 30 15:59:31.430466 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.430444 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" path="/var/lib/kubelet/pods/80a4faf0-b580-4e3b-b71c-cda031c990f9/volumes" Oct 30 15:59:31.431266 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.431244 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" path="/var/lib/kubelet/pods/b364f9b9-6f62-4bd2-b1f8-9636e024529a/volumes" Oct 30 15:59:31.724114 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.724009 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd"] Oct 30 15:59:31.724323 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.724290 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" containerID="cri-o://50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d" gracePeriod=30 Oct 30 15:59:31.738119 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.738083 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7"] Oct 30 15:59:31.738367 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.738343 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" containerID="cri-o://2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51" gracePeriod=30 Oct 30 15:59:31.761974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.756227 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b"] Oct 30 15:59:31.761974 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.756474 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" containerID="cri-o://c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da" gracePeriod=30 Oct 30 15:59:31.785806 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.785770 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw"] Oct 30 15:59:31.786016 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:31.785995 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" containerID="cri-o://10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248" gracePeriod=30 Oct 30 15:59:32.115479 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.113520 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:59:32.128169 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.128133 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:59:32.128334 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.128189 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 15:59:32.216959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.216922 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb734159-1499-4633-9a9e-aefdfcf9b76f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb734159-1499-4633-9a9e-aefdfcf9b76f-istiod-ca-cert\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217235 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.216982 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-kube-api-access-6vn4h\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-kube-api-access-6vn4h\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217235 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217189 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-podinfo\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217281 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-envoy\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217369 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217331 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-token\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-token\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217473 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217372 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-data\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217463 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb734159-1499-4633-9a9e-aefdfcf9b76f-istiod-ca-cert" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:32.217532 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217469 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-certs\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217636 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217593 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-socket\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.217693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.217646 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-credential-socket\") pod \"fb734159-1499-4633-9a9e-aefdfcf9b76f\" (UID: \"fb734159-1499-4633-9a9e-aefdfcf9b76f\") " Oct 30 15:59:32.218234 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.218165 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-data" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.218234 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.218184 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-certs" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.218432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.218372 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-credential-socket" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.218432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.218383 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-socket" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.218432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.218380 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.218432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.218415 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.218432 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.218430 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb734159-1499-4633-9a9e-aefdfcf9b76f-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.220340 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.220307 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-envoy" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.220516 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.220494 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-token" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.220812 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.220788 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-kube-api-access-6vn4h" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "kube-api-access-6vn4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.222925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.222901 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-podinfo" pod "fb734159-1499-4633-9a9e-aefdfcf9b76f" (UID: "fb734159-1499-4633-9a9e-aefdfcf9b76f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:32.229661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.229590 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:59:32.232074 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.232053 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:59:32.303389 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.303350 4401 generic.go:356] "Generic (PLEG): container finished" podID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerID="ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3" exitCode=0 Oct 30 15:59:32.303573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.303420 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" Oct 30 15:59:32.303573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.303436 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" event={"ID":"dda4138b-2339-4c0f-a8f7-31d535aec102","Type":"ContainerDied","Data":"ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3"} Oct 30 15:59:32.303573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.303522 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm" event={"ID":"dda4138b-2339-4c0f-a8f7-31d535aec102","Type":"ContainerDied","Data":"f5690e4ad06fb1f8dc3e9ed8326490efb72ca34b03f67ab5c48bc31704c7e71e"} Oct 30 15:59:32.303573 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.303556 4401 scope.go:122] "RemoveContainer" containerID="ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3" Oct 30 15:59:32.305638 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.305606 4401 generic.go:356] "Generic (PLEG): container finished" podID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerID="48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49" exitCode=0 Oct 30 15:59:32.305772 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.305651 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" Oct 30 15:59:32.305772 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.305656 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" event={"ID":"fb734159-1499-4633-9a9e-aefdfcf9b76f","Type":"ContainerDied","Data":"48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49"} Oct 30 15:59:32.305772 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.305707 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj" event={"ID":"fb734159-1499-4633-9a9e-aefdfcf9b76f","Type":"ContainerDied","Data":"4ac05b280756b1dfb097be8609382e10ad61a7ab9c750048c47288d30e50a192"} Oct 30 15:59:32.307413 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.307393 4401 generic.go:356] "Generic (PLEG): container finished" podID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerID="637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd" exitCode=0 Oct 30 15:59:32.307859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.307458 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" Oct 30 15:59:32.307859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.307463 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" event={"ID":"be71f92e-9b24-4e70-911d-a6b62e659d56","Type":"ContainerDied","Data":"637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd"} Oct 30 15:59:32.307859 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.307569 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd" event={"ID":"be71f92e-9b24-4e70-911d-a6b62e659d56","Type":"ContainerDied","Data":"79471d652f5c06fee489d32dbde67f358cb8b67ae0b2e03556d11565d881c0a9"} Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319007 4401 scope.go:122] "RemoveContainer" containerID="ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319315 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-socket\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319367 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-envoy\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319412 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-kube-api-access-cldp8\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-kube-api-access-cldp8\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319470 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-data\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:32.319410 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3\": container with ID starting with ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3 not found: ID does not exist" containerID="ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319524 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-podinfo\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319551 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3"} err="failed to get container status \"ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3\": rpc error: code = NotFound desc = could not find container \"ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3\": container with ID starting with ed13e55b0306ec8688d58141680957d1caa89fe187c3f722afd5ba86e1ac19e3 not found: ID does not exist" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319582 4401 scope.go:122] "RemoveContainer" containerID="48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319608 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-socket" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319565 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-credential-socket\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319741 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be71f92e-9b24-4e70-911d-a6b62e659d56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be71f92e-9b24-4e70-911d-a6b62e659d56-istiod-ca-cert\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319786 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-token\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-token\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319796 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-data" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319861 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-socket\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319898 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-credential-socket\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319929 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-kube-api-access-94tw6\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-kube-api-access-94tw6\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319960 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-envoy\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.319993 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-certs\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320048 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-podinfo\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320081 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda4138b-2339-4c0f-a8f7-31d535aec102-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4138b-2339-4c0f-a8f7-31d535aec102-istiod-ca-cert\") pod \"dda4138b-2339-4c0f-a8f7-31d535aec102\" (UID: \"dda4138b-2339-4c0f-a8f7-31d535aec102\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320111 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-token\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-token\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320180 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-certs\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320233 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-data\") pod \"be71f92e-9b24-4e70-911d-a6b62e659d56\" (UID: \"be71f92e-9b24-4e70-911d-a6b62e659d56\") " Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320548 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-credential-socket" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320764 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-data" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320889 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320909 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320922 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320937 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320953 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb734159-1499-4633-9a9e-aefdfcf9b76f-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320966 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320980 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.320993 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vn4h\" (UniqueName: \"kubernetes.io/projected/fb734159-1499-4633-9a9e-aefdfcf9b76f-kube-api-access-6vn4h\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.321108 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.321006 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb734159-1499-4633-9a9e-aefdfcf9b76f-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.321561 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be71f92e-9b24-4e70-911d-a6b62e659d56-istiod-ca-cert" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.321567 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-certs" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.321773 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-socket" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.321977 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-credential-socket" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.322042 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-envoy" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.322205 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda4138b-2339-4c0f-a8f7-31d535aec102-istiod-ca-cert" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.323029 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-kube-api-access-cldp8" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "kube-api-access-cldp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.323044 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-token" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.323329 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.323285 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-certs" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.334546 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.334516 4401 scope.go:122] "RemoveContainer" containerID="beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b" Oct 30 15:59:32.335299 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.335266 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-kube-api-access-94tw6" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "kube-api-access-94tw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.378972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.345282 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-podinfo" pod "dda4138b-2339-4c0f-a8f7-31d535aec102" (UID: "dda4138b-2339-4c0f-a8f7-31d535aec102"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:32.378972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.345566 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-token" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.378972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.345612 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-envoy" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.378972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.376351 4401 scope.go:122] "RemoveContainer" containerID="48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49" Oct 30 15:59:32.378972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.376548 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-podinfo" pod "be71f92e-9b24-4e70-911d-a6b62e659d56" (UID: "be71f92e-9b24-4e70-911d-a6b62e659d56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:32.379998 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:32.379088 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49\": container with ID starting with 48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49 not found: ID does not exist" containerID="48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49" Oct 30 15:59:32.379998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.379148 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49"} err="failed to get container status \"48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49\": rpc error: code = NotFound desc = could not find container \"48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49\": container with ID starting with 48eda8d05c99bf1e9fd71892bf3d3b018b86512b913a822f5d1e78c015d41e49 not found: ID does not exist" Oct 30 15:59:32.379998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.379175 4401 scope.go:122] "RemoveContainer" containerID="beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b" Oct 30 15:59:32.379998 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:32.379592 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b\": container with ID starting with beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b not found: ID does not exist" containerID="beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b" Oct 30 15:59:32.379998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.379624 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b"} err="failed to get container status \"beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b\": rpc error: code = NotFound desc = could not find container \"beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b\": container with ID starting with beb8b50a406fe75e9414b139643c00274411057145d0a49f62ca1b097efb365b not found: ID does not exist" Oct 30 15:59:32.379998 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.379642 4401 scope.go:122] "RemoveContainer" containerID="637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd" Oct 30 15:59:32.399067 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.399031 4401 scope.go:122] "RemoveContainer" containerID="637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd" Oct 30 15:59:32.399461 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:32.399434 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd\": container with ID starting with 637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd not found: ID does not exist" containerID="637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd" Oct 30 15:59:32.399557 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.399473 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd"} err="failed to get container status \"637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd\": rpc error: code = NotFound desc = could not find container \"637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd\": container with ID starting with 637cc8b64a97e944aee81c875bfef0bfc58b03a97c1a3706ac4a31e06cf21afd not found: ID does not exist" Oct 30 15:59:32.422345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422304 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422340 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be71f92e-9b24-4e70-911d-a6b62e659d56-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422353 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422364 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422375 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422388 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94tw6\" (UniqueName: \"kubernetes.io/projected/dda4138b-2339-4c0f-a8f7-31d535aec102-kube-api-access-94tw6\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422398 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422410 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422423 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422434 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4138b-2339-4c0f-a8f7-31d535aec102-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422444 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422455 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422466 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be71f92e-9b24-4e70-911d-a6b62e659d56-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422477 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4138b-2339-4c0f-a8f7-31d535aec102-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.422645 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.422488 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cldp8\" (UniqueName: \"kubernetes.io/projected/be71f92e-9b24-4e70-911d-a6b62e659d56-kube-api-access-cldp8\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.476983 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.476945 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj"] Oct 30 15:59:32.481945 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.481875 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-z4fxj"] Oct 30 15:59:32.699654 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.696131 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd"] Oct 30 15:59:32.702681 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.702641 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-wx7wd"] Oct 30 15:59:32.732183 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.732092 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm"] Oct 30 15:59:32.738905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.738870 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2b3f1-openshift-default-57587c5fc-nl8tm"] Oct 30 15:59:32.890542 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.890510 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:59:32.927545 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927516 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istiod-ca-cert\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.927646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927592 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-certs\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.927646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927626 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-token\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-token\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.927762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927655 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-envoy\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.927762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927702 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-credential-socket\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.927762 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927749 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-podinfo\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.927899 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927804 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-data\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.927899 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927890 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-socket\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.928010 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927955 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-kube-api-access-4tlrq\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-kube-api-access-4tlrq\") pod \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\" (UID: \"2d546ec3-604a-45bc-9a22-5a3e67f069e8\") " Oct 30 15:59:32.928064 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.927994 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istiod-ca-cert" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:32.928348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.928317 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-socket" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.928348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.928334 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:32.928491 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.928114 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-certs" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.928763 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.928739 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-data" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.928925 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.928879 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-credential-socket" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.930619 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.930592 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-envoy" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:32.930858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.930797 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-kube-api-access-4tlrq" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "kube-api-access-4tlrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.930981 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.930912 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-token" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:32.930981 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.930930 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-podinfo" pod "2d546ec3-604a-45bc-9a22-5a3e67f069e8" (UID: "2d546ec3-604a-45bc-9a22-5a3e67f069e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:32.969851 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:32.969228 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:59:33.028794 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.028703 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-data\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.028794 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.028767 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-credential-socket\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.028803 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-envoy\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.028946 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-socket\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.028975 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-token\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-token\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.029004 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb54a07-e06c-4575-8d85-cf9bb779283f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb54a07-e06c-4575-8d85-cf9bb779283f-istiod-ca-cert\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029056 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.029045 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-kube-api-access-w7nps\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-kube-api-access-w7nps\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.029114 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-certs\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.029163 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-podinfo\") pod \"0bb54a07-e06c-4575-8d85-cf9bb779283f\" (UID: \"0bb54a07-e06c-4575-8d85-cf9bb779283f\") " Oct 30 15:59:33.029358 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.029312 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-data" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030350 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030382 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030396 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030408 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030425 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030438 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tlrq\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-kube-api-access-4tlrq\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030452 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030465 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030478 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d546ec3-604a-45bc-9a22-5a3e67f069e8-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030402 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-credential-socket" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:33.030520 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030415 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-socket" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:33.031238 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.030613 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb54a07-e06c-4575-8d85-cf9bb779283f-istiod-ca-cert" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:33.031457 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.031430 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-certs" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:33.033113 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.033082 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-kube-api-access-w7nps" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "kube-api-access-w7nps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:33.033384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.033358 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-envoy" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:33.033384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.033357 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-podinfo" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:33.033680 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.033656 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-token" pod "0bb54a07-e06c-4575-8d85-cf9bb779283f" (UID: "0bb54a07-e06c-4575-8d85-cf9bb779283f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:33.124646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.124604 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:59:33.124804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.124674 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 15:59:33.131293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131262 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.131293 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131292 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.131462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131302 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.131462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131311 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.131462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131319 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bb54a07-e06c-4575-8d85-cf9bb779283f-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.131462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131327 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nps\" (UniqueName: \"kubernetes.io/projected/0bb54a07-e06c-4575-8d85-cf9bb779283f-kube-api-access-w7nps\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.131462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131335 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bb54a07-e06c-4575-8d85-cf9bb779283f-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.131462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.131343 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bb54a07-e06c-4575-8d85-cf9bb779283f-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:33.317408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.317310 4401 generic.go:356] "Generic (PLEG): container finished" podID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerID="00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491" exitCode=0 Oct 30 15:59:33.317408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.317372 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" Oct 30 15:59:33.317408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.317384 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" event={"ID":"0bb54a07-e06c-4575-8d85-cf9bb779283f","Type":"ContainerDied","Data":"00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491"} Oct 30 15:59:33.317661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.317480 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm" event={"ID":"0bb54a07-e06c-4575-8d85-cf9bb779283f","Type":"ContainerDied","Data":"b5efb8cf9f4d9ec208cd15ee28d015f813249aa01841f0899060c9f7ddd5a955"} Oct 30 15:59:33.317661 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.317515 4401 scope.go:122] "RemoveContainer" containerID="00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491" Oct 30 15:59:33.321315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.321267 4401 generic.go:356] "Generic (PLEG): container finished" podID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerID="1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327" exitCode=0 Oct 30 15:59:33.321315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.321301 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" event={"ID":"2d546ec3-604a-45bc-9a22-5a3e67f069e8","Type":"ContainerDied","Data":"1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327"} Oct 30 15:59:33.321315 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.321316 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" Oct 30 15:59:33.321558 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.321358 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7" event={"ID":"2d546ec3-604a-45bc-9a22-5a3e67f069e8","Type":"ContainerDied","Data":"763ca8d87eb00d4b16893dc302664721cf6d5b05045869da1e6996f7e734ea9a"} Oct 30 15:59:33.330514 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.330350 4401 scope.go:122] "RemoveContainer" containerID="00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491" Oct 30 15:59:33.330896 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:33.330872 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491\": container with ID starting with 00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491 not found: ID does not exist" containerID="00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491" Oct 30 15:59:33.331001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.330905 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491"} err="failed to get container status \"00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491\": rpc error: code = NotFound desc = could not find container \"00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491\": container with ID starting with 00e6f3dabdd99d1af8a8fff9b5a0671f9c81209ae8cf5d916dbac135a7fe8491 not found: ID does not exist" Oct 30 15:59:33.331001 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.330924 4401 scope.go:122] "RemoveContainer" containerID="1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327" Oct 30 15:59:33.342301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.342274 4401 scope.go:122] "RemoveContainer" containerID="1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327" Oct 30 15:59:33.342640 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:33.342619 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327\": container with ID starting with 1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327 not found: ID does not exist" containerID="1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327" Oct 30 15:59:33.342685 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.342651 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327"} err="failed to get container status \"1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327\": rpc error: code = NotFound desc = could not find container \"1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327\": container with ID starting with 1cbf59675d9d752ea67eb9076014a97dfcc5f70e49efd63dffad82b2fb8fa327 not found: ID does not exist" Oct 30 15:59:33.395667 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.395624 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7"] Oct 30 15:59:33.400592 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.400560 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-ntql7"] Oct 30 15:59:33.428202 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.428168 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" path="/var/lib/kubelet/pods/2d546ec3-604a-45bc-9a22-5a3e67f069e8/volumes" Oct 30 15:59:33.428648 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.428634 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" path="/var/lib/kubelet/pods/be71f92e-9b24-4e70-911d-a6b62e659d56/volumes" Oct 30 15:59:33.429088 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.429074 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" path="/var/lib/kubelet/pods/dda4138b-2339-4c0f-a8f7-31d535aec102/volumes" Oct 30 15:59:33.429908 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.429890 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" path="/var/lib/kubelet/pods/fb734159-1499-4633-9a9e-aefdfcf9b76f/volumes" Oct 30 15:59:33.439115 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.439085 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm"] Oct 30 15:59:33.443948 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:33.443926 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-040f9a-openshift-default-6bd6df64dc-v5xwm"] Oct 30 15:59:34.124183 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:34.124148 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:59:34.124588 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:34.124203 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 15:59:35.109700 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.109659 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb"] Oct 30 15:59:35.110189 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110175 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" Oct 30 15:59:35.110231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110190 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" Oct 30 15:59:35.110231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110210 4401 cpu_manager.go:414] "containerMap: removing container" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" Oct 30 15:59:35.110231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110215 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" Oct 30 15:59:35.110231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110226 4401 cpu_manager.go:414] "containerMap: removing container" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" Oct 30 15:59:35.110231 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110231 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110244 4401 cpu_manager.go:414] "containerMap: removing container" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110249 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110255 4401 cpu_manager.go:414] "containerMap: removing container" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110259 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110274 4401 cpu_manager.go:414] "containerMap: removing container" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110280 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110290 4401 cpu_manager.go:414] "containerMap: removing container" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110295 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110303 4401 cpu_manager.go:414] "containerMap: removing container" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110308 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110318 4401 cpu_manager.go:414] "containerMap: removing container" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110322 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110327 4401 cpu_manager.go:414] "containerMap: removing container" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110333 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110339 4401 cpu_manager.go:414] "containerMap: removing container" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" Oct 30 15:59:35.110404 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110343 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110418 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="be71f92e-9b24-4e70-911d-a6b62e659d56" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110429 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c5b157-a5fd-4560-b25f-25719b24fef7" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110436 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110442 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb734159-1499-4633-9a9e-aefdfcf9b76f" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110448 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cdc8085-eee2-4c42-9d59-8935888b5d65" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110455 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d546ec3-604a-45bc-9a22-5a3e67f069e8" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110461 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110468 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110473 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda4138b-2339-4c0f-a8f7-31d535aec102" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110477 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a4faf0-b580-4e3b-b71c-cda031c990f9" containerName="istio-proxy" Oct 30 15:59:35.110858 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.110482 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="b364f9b9-6f62-4bd2-b1f8-9636e024529a" containerName="istio-proxy" Oct 30 15:59:35.115658 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.115634 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.124997 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.124963 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:59:35.125428 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.125067 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 15:59:35.134959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.134919 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb"] Oct 30 15:59:35.139583 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.139547 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 15:59:35.139748 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.139612 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:59:35.140611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.140586 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:59:35.140719 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.140586 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:59:35.140719 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.140695 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:59:35.146651 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.146625 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:59:35.150991 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.150933 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bb44\" (UniqueName: \"kubernetes.io/projected/c6c93106-f3e9-4461-ae1a-e64e99e282b7-kube-api-access-6bb44\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.151196 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.151165 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.151331 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.151241 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.252191 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.252144 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.252345 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.252247 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.252384 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.252357 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bb44\" (UniqueName: \"kubernetes.io/projected/c6c93106-f3e9-4461-ae1a-e64e99e282b7-kube-api-access-6bb44\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.252813 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.252790 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.254722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.254692 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.278224 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.278189 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bb44\" (UniqueName: \"kubernetes.io/projected/c6c93106-f3e9-4461-ae1a-e64e99e282b7-kube-api-access-6bb44\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.427580 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.427490 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb54a07-e06c-4575-8d85-cf9bb779283f" path="/var/lib/kubelet/pods/0bb54a07-e06c-4575-8d85-cf9bb779283f/volumes" Oct 30 15:59:35.431314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.431283 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:35.745191 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:59:35.745147 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c93106_f3e9_4461_ae1a_e64e99e282b7.slice/crio-781543423f1abdea17910eb7350cc5b6b7104c5c6d9d8df5a569072e71072290 WatchSource:0}: Error finding container 781543423f1abdea17910eb7350cc5b6b7104c5c6d9d8df5a569072e71072290: Status 404 returned error can't find the container with id 781543423f1abdea17910eb7350cc5b6b7104c5c6d9d8df5a569072e71072290 Oct 30 15:59:35.749239 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:35.749207 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb"] Oct 30 15:59:36.162631 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.162599 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:36.163125 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.163093 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:36.165004 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.164966 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 15:59:36.334958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.334871 4401 patch_prober.go:29] interesting pod/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:59:36.334958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.334938 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:59:36.355730 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.355685 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" event={"ID":"c6c93106-f3e9-4461-ae1a-e64e99e282b7","Type":"ContainerStarted","Data":"e2ab850d7fce3a7f7f85b5a21328d3559328d49e0a0e7f2b1b9dfe972635af5c"} Oct 30 15:59:36.355909 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.355756 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" event={"ID":"c6c93106-f3e9-4461-ae1a-e64e99e282b7","Type":"ContainerStarted","Data":"781543423f1abdea17910eb7350cc5b6b7104c5c6d9d8df5a569072e71072290"} Oct 30 15:59:36.410380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:36.410307 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" podStartSLOduration=1.410288089 podStartE2EDuration="1.410288089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:59:36.404685392 +0000 UTC m=+1285.103958577" watchObservedRunningTime="2026-10-30 15:59:36.410288089 +0000 UTC m=+1285.109561247" Oct 30 15:59:37.059325 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.059292 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:59:37.086652 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.086608 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:59:37.136254 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.136222 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:59:37.154958 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.154926 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:59:37.174338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174295 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-data\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.174338 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174338 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-kube-api-access-tnqwd\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-kube-api-access-tnqwd\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174359 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-socket\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174379 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-credential-socket\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174476 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-token\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-token\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174513 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-credential-socket\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174540 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-data\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174566 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-certs\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174625 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-data" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174634 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-socket" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174633 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-envoy\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174692 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-certs\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174724 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-token\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-token\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174762 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-socket\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.174941 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.174928 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-data" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175004 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-certs" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175090 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-credential-socket" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175140 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-credential-socket" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175142 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-data\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175162 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-socket" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175211 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istiod-ca-cert\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175239 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-socket\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175258 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-kube-api-access-68rqj\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-kube-api-access-68rqj\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175323 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-envoy\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175330 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-data" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175395 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-kube-api-access-549lc\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-kube-api-access-549lc\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175424 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-token\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-token\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175461 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f6b2e0-696d-49df-81a7-f833d63e14ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6b2e0-696d-49df-81a7-f833d63e14ee-istiod-ca-cert\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175507 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-certs\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175559 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-podinfo\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175569 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istiod-ca-cert" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175591 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-podinfo\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175626 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-podinfo\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175654 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-envoy\") pod \"72f6b2e0-696d-49df-81a7-f833d63e14ee\" (UID: \"72f6b2e0-696d-49df-81a7-f833d63e14ee\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175686 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a30a03f0-6167-47ce-8897-47c6de0bfae8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30a03f0-6167-47ce-8897-47c6de0bfae8-istiod-ca-cert\") pod \"a30a03f0-6167-47ce-8897-47c6de0bfae8\" (UID: \"a30a03f0-6167-47ce-8897-47c6de0bfae8\") " Oct 30 15:59:37.175720 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175720 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-credential-socket\") pod \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\" (UID: \"fdba6667-d2cb-41d3-bafb-e0d04c8fc808\") " Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175856 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-socket" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.175936 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f6b2e0-696d-49df-81a7-f833d63e14ee-istiod-ca-cert" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176119 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-credential-socket" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176326 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176347 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176362 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176375 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176392 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176413 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176429 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176441 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f6b2e0-696d-49df-81a7-f833d63e14ee-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176455 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176469 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176483 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.176803 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.176498 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.177694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.177148 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-certs" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.177694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.177397 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-certs" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.177694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.177442 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a30a03f0-6167-47ce-8897-47c6de0bfae8-istiod-ca-cert" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:37.177694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.177511 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-kube-api-access-tnqwd" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "kube-api-access-tnqwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.177694 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.177564 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-token" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.179394 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.179360 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-token" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.180120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.180009 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-envoy" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.180120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.180022 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-envoy" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.180120 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.180087 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-kube-api-access-68rqj" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "kube-api-access-68rqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.180348 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.180155 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-token" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.180905 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.180874 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-podinfo" pod "a30a03f0-6167-47ce-8897-47c6de0bfae8" (UID: "a30a03f0-6167-47ce-8897-47c6de0bfae8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:37.181205 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.181180 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-envoy" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.181591 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.181564 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-podinfo" pod "fdba6667-d2cb-41d3-bafb-e0d04c8fc808" (UID: "fdba6667-d2cb-41d3-bafb-e0d04c8fc808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:37.182118 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.182082 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-kube-api-access-549lc" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "kube-api-access-549lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.182474 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.182444 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-podinfo" pod "72f6b2e0-696d-49df-81a7-f833d63e14ee" (UID: "72f6b2e0-696d-49df-81a7-f833d63e14ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:37.277159 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277119 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-kube-api-access-cqmbr\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-kube-api-access-cqmbr\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277167 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istiod-ca-cert\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277203 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-podinfo\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277318 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277232 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-certs\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277496 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277461 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-credential-socket\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277547 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277513 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-envoy\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277598 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277562 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-token\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-token\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277589 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-certs" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.277644 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277607 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-socket\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277660 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-data\") pod \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\" (UID: \"564bbe1a-6e89-4b9f-9432-795b6205f0a8\") " Oct 30 15:59:37.277754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277683 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-credential-socket" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.277754 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.277703 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istiod-ca-cert" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:37.278145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278118 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-socket" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.278145 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278133 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-data" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.278292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278209 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278230 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278245 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rqj\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-kube-api-access-68rqj\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278259 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278273 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278292 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278289 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-549lc\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-kube-api-access-549lc\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278307 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278321 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278335 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278347 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278360 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278373 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f6b2e0-696d-49df-81a7-f833d63e14ee-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278386 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30a03f0-6167-47ce-8897-47c6de0bfae8-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278399 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnqwd\" (UniqueName: \"kubernetes.io/projected/fdba6667-d2cb-41d3-bafb-e0d04c8fc808-kube-api-access-tnqwd\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278410 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30a03f0-6167-47ce-8897-47c6de0bfae8-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278423 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30a03f0-6167-47ce-8897-47c6de0bfae8-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278434 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.278611 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.278446 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.280040 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.280007 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-podinfo" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:59:37.280470 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.280441 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-token" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.280565 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.280449 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-envoy" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:37.281388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.281361 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-kube-api-access-cqmbr" pod "564bbe1a-6e89-4b9f-9432-795b6205f0a8" (UID: "564bbe1a-6e89-4b9f-9432-795b6205f0a8"). InnerVolumeSpecName "kube-api-access-cqmbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:37.366271 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.366236 4401 generic.go:356] "Generic (PLEG): container finished" podID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerID="50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d" exitCode=0 Oct 30 15:59:37.366456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.366294 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" Oct 30 15:59:37.366456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.366325 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" event={"ID":"564bbe1a-6e89-4b9f-9432-795b6205f0a8","Type":"ContainerDied","Data":"50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d"} Oct 30 15:59:37.366456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.366379 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd" event={"ID":"564bbe1a-6e89-4b9f-9432-795b6205f0a8","Type":"ContainerDied","Data":"fbcb1a367744fa7fb8e02b3a387cdf1574e4a36bd61a4469368316567b71e879"} Oct 30 15:59:37.366456 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.366401 4401 scope.go:122] "RemoveContainer" containerID="50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d" Oct 30 15:59:37.368380 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.368354 4401 generic.go:356] "Generic (PLEG): container finished" podID="c6c93106-f3e9-4461-ae1a-e64e99e282b7" containerID="e2ab850d7fce3a7f7f85b5a21328d3559328d49e0a0e7f2b1b9dfe972635af5c" exitCode=0 Oct 30 15:59:37.368492 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.368426 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" event={"ID":"c6c93106-f3e9-4461-ae1a-e64e99e282b7","Type":"ContainerDied","Data":"e2ab850d7fce3a7f7f85b5a21328d3559328d49e0a0e7f2b1b9dfe972635af5c"} Oct 30 15:59:37.370015 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.369982 4401 generic.go:356] "Generic (PLEG): container finished" podID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerID="10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248" exitCode=0 Oct 30 15:59:37.370127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.370041 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" Oct 30 15:59:37.370127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.370067 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" event={"ID":"fdba6667-d2cb-41d3-bafb-e0d04c8fc808","Type":"ContainerDied","Data":"10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248"} Oct 30 15:59:37.370127 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.370119 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw" event={"ID":"fdba6667-d2cb-41d3-bafb-e0d04c8fc808","Type":"ContainerDied","Data":"112016c8f9a1f522eecf196a6cd4bcfd7ec5c0e462bfef424bf36cd413304cca"} Oct 30 15:59:37.371710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.371682 4401 generic.go:356] "Generic (PLEG): container finished" podID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerID="c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da" exitCode=0 Oct 30 15:59:37.371819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.371731 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" Oct 30 15:59:37.371819 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.371760 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" event={"ID":"72f6b2e0-696d-49df-81a7-f833d63e14ee","Type":"ContainerDied","Data":"c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da"} Oct 30 15:59:37.371972 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.371812 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b" event={"ID":"72f6b2e0-696d-49df-81a7-f833d63e14ee","Type":"ContainerDied","Data":"8105aa91c049ef535e8e949b4bb719e21f1e915956b1bb3cc0d2df401f334e6d"} Oct 30 15:59:37.374219 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.374190 4401 generic.go:356] "Generic (PLEG): container finished" podID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerID="2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51" exitCode=0 Oct 30 15:59:37.374344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.374246 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" Oct 30 15:59:37.374344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.374271 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" event={"ID":"a30a03f0-6167-47ce-8897-47c6de0bfae8","Type":"ContainerDied","Data":"2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51"} Oct 30 15:59:37.374344 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.374320 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7" event={"ID":"a30a03f0-6167-47ce-8897-47c6de0bfae8","Type":"ContainerDied","Data":"93d370ddb0f403b6571ee4c451071a0dec26d207376de4ce7ca8c0e989452f27"} Oct 30 15:59:37.379005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.378981 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.379005 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.379003 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.379184 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.379012 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.379184 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.379022 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqmbr\" (UniqueName: \"kubernetes.io/projected/564bbe1a-6e89-4b9f-9432-795b6205f0a8-kube-api-access-cqmbr\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.379184 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.379031 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.379184 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.379039 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/564bbe1a-6e89-4b9f-9432-795b6205f0a8-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:37.389264 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.380963 4401 scope.go:122] "RemoveContainer" containerID="cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988" Oct 30 15:59:37.401242 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.401210 4401 scope.go:122] "RemoveContainer" containerID="50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d" Oct 30 15:59:37.401598 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:37.401576 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d\": container with ID starting with 50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d not found: ID does not exist" containerID="50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d" Oct 30 15:59:37.401659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.401609 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d"} err="failed to get container status \"50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d\": rpc error: code = NotFound desc = could not find container \"50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d\": container with ID starting with 50cff2c84e9ebef0b1226353833763194ef7e3a9322069ba66f73d591afb771d not found: ID does not exist" Oct 30 15:59:37.401659 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.401626 4401 scope.go:122] "RemoveContainer" containerID="cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988" Oct 30 15:59:37.401905 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:37.401884 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988\": container with ID starting with cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988 not found: ID does not exist" containerID="cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988" Oct 30 15:59:37.401978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.401915 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988"} err="failed to get container status \"cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988\": rpc error: code = NotFound desc = could not find container \"cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988\": container with ID starting with cbba60ed6f8d61c39103667ac3b4f39a5d2a0a55b1e01713cfd77b04d4f86988 not found: ID does not exist" Oct 30 15:59:37.401978 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.401936 4401 scope.go:122] "RemoveContainer" containerID="10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248" Oct 30 15:59:37.412607 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.412580 4401 scope.go:122] "RemoveContainer" containerID="10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248" Oct 30 15:59:37.412975 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:37.412951 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248\": container with ID starting with 10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248 not found: ID does not exist" containerID="10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248" Oct 30 15:59:37.413055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.412984 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248"} err="failed to get container status \"10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248\": rpc error: code = NotFound desc = could not find container \"10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248\": container with ID starting with 10312e2c69af3ecbf935291e5a617ce50a295736e944329ace53b64b52c93248 not found: ID does not exist" Oct 30 15:59:37.413055 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.413003 4401 scope.go:122] "RemoveContainer" containerID="c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da" Oct 30 15:59:37.423959 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.423931 4401 scope.go:122] "RemoveContainer" containerID="c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da" Oct 30 15:59:37.424349 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:37.424315 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da\": container with ID starting with c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da not found: ID does not exist" containerID="c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da" Oct 30 15:59:37.424489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.424357 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da"} err="failed to get container status \"c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da\": rpc error: code = NotFound desc = could not find container \"c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da\": container with ID starting with c4d2cde48320881043bdb6573f4674821c8d6d18b5247ab260d3f54359ed87da not found: ID does not exist" Oct 30 15:59:37.424489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.424381 4401 scope.go:122] "RemoveContainer" containerID="2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51" Oct 30 15:59:37.438865 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.438811 4401 scope.go:122] "RemoveContainer" containerID="2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51" Oct 30 15:59:37.439225 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:37.439202 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51\": container with ID starting with 2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51 not found: ID does not exist" containerID="2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51" Oct 30 15:59:37.439301 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.439239 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51"} err="failed to get container status \"2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51\": rpc error: code = NotFound desc = could not find container \"2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51\": container with ID starting with 2660c285bec0c3a32ad0e194137f7abc2036c23156b4ff40b11673f50ce66a51 not found: ID does not exist" Oct 30 15:59:37.494712 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.494677 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd"] Oct 30 15:59:37.499465 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.499434 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-488zd"] Oct 30 15:59:37.575306 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.575277 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7"] Oct 30 15:59:37.579901 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.579863 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-j5mw7"] Oct 30 15:59:37.616969 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.616936 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw"] Oct 30 15:59:37.623260 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.623231 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-nbfdw"] Oct 30 15:59:37.676395 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.676314 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b"] Oct 30 15:59:37.682646 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:37.682595 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-9e2b29-openshift-default-7cb44dddb7-pk72b"] Oct 30 15:59:38.534185 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.534155 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:38.594634 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.594593 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-openshift-ca\") pod \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " Oct 30 15:59:38.594810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.594664 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c93106-f3e9-4461-ae1a-e64e99e282b7-kube-api-access-6bb44\" (UniqueName: \"kubernetes.io/projected/c6c93106-f3e9-4461-ae1a-e64e99e282b7-kube-api-access-6bb44\") pod \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " Oct 30 15:59:38.594810 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.594784 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-xks-ca\") pod \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\" (UID: \"c6c93106-f3e9-4461-ae1a-e64e99e282b7\") " Oct 30 15:59:38.595171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.595019 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-openshift-ca" pod "c6c93106-f3e9-4461-ae1a-e64e99e282b7" (UID: "c6c93106-f3e9-4461-ae1a-e64e99e282b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:38.595171 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.595150 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:38.596995 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.596963 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c93106-f3e9-4461-ae1a-e64e99e282b7-kube-api-access-6bb44" pod "c6c93106-f3e9-4461-ae1a-e64e99e282b7" (UID: "c6c93106-f3e9-4461-ae1a-e64e99e282b7"). InnerVolumeSpecName "kube-api-access-6bb44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:38.597148 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.597113 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-xks-ca" pod "c6c93106-f3e9-4461-ae1a-e64e99e282b7" (UID: "c6c93106-f3e9-4461-ae1a-e64e99e282b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:38.695876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.695776 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bb44\" (UniqueName: \"kubernetes.io/projected/c6c93106-f3e9-4461-ae1a-e64e99e282b7-kube-api-access-6bb44\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:38.695876 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:38.695811 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6c93106-f3e9-4461-ae1a-e64e99e282b7-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:39.396224 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.396195 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" Oct 30 15:59:39.396388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.396233 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb" event={"ID":"c6c93106-f3e9-4461-ae1a-e64e99e282b7","Type":"ContainerDied","Data":"781543423f1abdea17910eb7350cc5b6b7104c5c6d9d8df5a569072e71072290"} Oct 30 15:59:39.396388 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.396284 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="781543423f1abdea17910eb7350cc5b6b7104c5c6d9d8df5a569072e71072290" Oct 30 15:59:39.428268 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.428238 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" path="/var/lib/kubelet/pods/564bbe1a-6e89-4b9f-9432-795b6205f0a8/volumes" Oct 30 15:59:39.428718 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.428704 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" path="/var/lib/kubelet/pods/72f6b2e0-696d-49df-81a7-f833d63e14ee/volumes" Oct 30 15:59:39.429147 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.429126 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" path="/var/lib/kubelet/pods/a30a03f0-6167-47ce-8897-47c6de0bfae8/volumes" Oct 30 15:59:39.429880 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.429865 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" path="/var/lib/kubelet/pods/fdba6667-d2cb-41d3-bafb-e0d04c8fc808/volumes" Oct 30 15:59:39.900400 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:39.900362 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2"] Oct 30 15:59:40.149695 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:40.149644 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 15:59:45.626722 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:45.626676 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h"] Oct 30 15:59:45.653272 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:45.653233 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9"] Oct 30 15:59:47.037112 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.037073 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw"] Oct 30 15:59:47.054507 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.054466 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1af72af3-79xbw"] Oct 30 15:59:47.088932 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.088892 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t"] Oct 30 15:59:47.089462 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089443 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089464 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089494 4401 cpu_manager.go:414] "containerMap: removing container" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089502 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089513 4401 cpu_manager.go:414] "containerMap: removing container" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089520 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089531 4401 cpu_manager.go:414] "containerMap: removing container" podUID="c6c93106-f3e9-4461-ae1a-e64e99e282b7" containerName="revoke-keys" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089538 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c93106-f3e9-4461-ae1a-e64e99e282b7" containerName="revoke-keys" Oct 30 15:59:47.089562 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089559 4401 cpu_manager.go:414] "containerMap: removing container" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089566 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089601 4401 cpu_manager.go:414] "containerMap: removing container" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089608 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089747 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089760 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f6b2e0-696d-49df-81a7-f833d63e14ee" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089772 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c93106-f3e9-4461-ae1a-e64e99e282b7" containerName="revoke-keys" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089783 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdba6667-d2cb-41d3-bafb-e0d04c8fc808" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089796 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="564bbe1a-6e89-4b9f-9432-795b6205f0a8" containerName="istio-proxy" Oct 30 15:59:47.089994 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.089806 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a30a03f0-6167-47ce-8897-47c6de0bfae8" containerName="istio-proxy" Oct 30 15:59:47.095220 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.095194 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.115693 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.115662 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t"] Oct 30 15:59:47.117236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.117115 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:59:47.117236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.117188 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 15:59:47.119041 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.119014 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:59:47.119818 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.119800 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:59:47.185412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.185351 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9475\" (UniqueName: \"kubernetes.io/projected/28f6e172-6f85-4f93-ba69-f9cf055dc901-kube-api-access-k9475\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.185614 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.185440 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.185674 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.185593 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.286245 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.286206 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9475\" (UniqueName: \"kubernetes.io/projected/28f6e172-6f85-4f93-ba69-f9cf055dc901-kube-api-access-k9475\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.286471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.286263 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.286471 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.286321 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.287153 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.287078 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.288996 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.288966 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.313408 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.313376 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9475\" (UniqueName: \"kubernetes.io/projected/28f6e172-6f85-4f93-ba69-f9cf055dc901-kube-api-access-k9475\") pod \"maas-api-revoke-sub-a89e2376-jx47t\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.425304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.425273 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:47.427690 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.427658 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="238833d8-5528-4d55-8e4a-df5ee798ff55" path="/var/lib/kubelet/pods/238833d8-5528-4d55-8e4a-df5ee798ff55/volumes" Oct 30 15:59:47.741145 ip-10-0-133-222 kubenswrapper[4401]: W1030 15:59:47.741102 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f6e172_6f85_4f93_ba69_f9cf055dc901.slice/crio-d998bed929b41aca0361b05731851e305541c6c6a48facde763713a33e9f4cb5 WatchSource:0}: Error finding container d998bed929b41aca0361b05731851e305541c6c6a48facde763713a33e9f4cb5: Status 404 returned error can't find the container with id d998bed929b41aca0361b05731851e305541c6c6a48facde763713a33e9f4cb5 Oct 30 15:59:47.744804 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:47.744770 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t"] Oct 30 15:59:48.481710 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:48.481668 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" event={"ID":"28f6e172-6f85-4f93-ba69-f9cf055dc901","Type":"ContainerStarted","Data":"d9af3519e915fbba0967ea4e9cbf106297cdd0dae2c04e8dd1682ca375ef7f1a"} Oct 30 15:59:48.482175 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:48.481764 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" event={"ID":"28f6e172-6f85-4f93-ba69-f9cf055dc901","Type":"ContainerStarted","Data":"d998bed929b41aca0361b05731851e305541c6c6a48facde763713a33e9f4cb5"} Oct 30 15:59:48.531711 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:48.531653 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" podStartSLOduration=1.53163668 podStartE2EDuration="1.53163668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:59:48.526759171 +0000 UTC m=+1297.226032329" watchObservedRunningTime="2026-10-30 15:59:48.53163668 +0000 UTC m=+1297.230909836" Oct 30 15:59:49.492087 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:49.492050 4401 generic.go:356] "Generic (PLEG): container finished" podID="28f6e172-6f85-4f93-ba69-f9cf055dc901" containerID="d9af3519e915fbba0967ea4e9cbf106297cdd0dae2c04e8dd1682ca375ef7f1a" exitCode=0 Oct 30 15:59:49.492477 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:49.492101 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" event={"ID":"28f6e172-6f85-4f93-ba69-f9cf055dc901","Type":"ContainerDied","Data":"d9af3519e915fbba0967ea4e9cbf106297cdd0dae2c04e8dd1682ca375ef7f1a"} Oct 30 15:59:50.627906 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.627866 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" podUID="72b2a776-bd23-41f3-908d-81cfc68ffb47" containerName="payload-processing" containerID="cri-o://8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c" gracePeriod=25 Oct 30 15:59:50.653612 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.653578 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:50.653771 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.653687 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" podUID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" containerName="payload-pre-processing" containerID="cri-o://5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e" gracePeriod=25 Oct 30 15:59:50.722314 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.722279 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-openshift-ca\") pod \"28f6e172-6f85-4f93-ba69-f9cf055dc901\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " Oct 30 15:59:50.722517 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.722333 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-xks-ca\") pod \"28f6e172-6f85-4f93-ba69-f9cf055dc901\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " Oct 30 15:59:50.722517 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.722431 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28f6e172-6f85-4f93-ba69-f9cf055dc901-kube-api-access-k9475\" (UniqueName: \"kubernetes.io/projected/28f6e172-6f85-4f93-ba69-f9cf055dc901-kube-api-access-k9475\") pod \"28f6e172-6f85-4f93-ba69-f9cf055dc901\" (UID: \"28f6e172-6f85-4f93-ba69-f9cf055dc901\") " Oct 30 15:59:50.722731 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.722704 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-openshift-ca" pod "28f6e172-6f85-4f93-ba69-f9cf055dc901" (UID: "28f6e172-6f85-4f93-ba69-f9cf055dc901"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:59:50.722869 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.722853 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:50.724626 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.724603 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-xks-ca" pod "28f6e172-6f85-4f93-ba69-f9cf055dc901" (UID: "28f6e172-6f85-4f93-ba69-f9cf055dc901"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:50.724704 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.724621 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f6e172-6f85-4f93-ba69-f9cf055dc901-kube-api-access-k9475" pod "28f6e172-6f85-4f93-ba69-f9cf055dc901" (UID: "28f6e172-6f85-4f93-ba69-f9cf055dc901"). InnerVolumeSpecName "kube-api-access-k9475". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:50.823605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.823511 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9475\" (UniqueName: \"kubernetes.io/projected/28f6e172-6f85-4f93-ba69-f9cf055dc901-kube-api-access-k9475\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:50.823605 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:50.823547 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28f6e172-6f85-4f93-ba69-f9cf055dc901-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:51.510090 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:51.510058 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" Oct 30 15:59:51.510304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:51.510091 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t" event={"ID":"28f6e172-6f85-4f93-ba69-f9cf055dc901","Type":"ContainerDied","Data":"d998bed929b41aca0361b05731851e305541c6c6a48facde763713a33e9f4cb5"} Oct 30 15:59:51.510304 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:51.510156 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d998bed929b41aca0361b05731851e305541c6c6a48facde763713a33e9f4cb5" Oct 30 15:59:51.967762 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:51.967721 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 15:59:54.920879 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:54.920759 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" podUID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" containerName="main" containerID="cri-o://742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f" gracePeriod=45 Oct 30 15:59:55.182206 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.182138 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:59:55.269274 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269241 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-kube-api-access-zcknw\" (UniqueName: \"kubernetes.io/projected/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-kube-api-access-zcknw\") pod \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " Oct 30 15:59:55.269484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269291 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-dshm\") pod \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " Oct 30 15:59:55.269484 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269420 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-model-cache\") pod \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " Oct 30 15:59:55.269606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269496 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-home\") pod \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " Oct 30 15:59:55.269606 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269569 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tmp-dir\") pod \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " Oct 30 15:59:55.269715 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269606 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tls-certs\") pod \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\" (UID: \"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf\") " Oct 30 15:59:55.269772 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269691 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-model-cache" pod "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" (UID: "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:55.269873 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269769 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-home" pod "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" (UID: "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:55.270007 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.269983 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tmp-dir" pod "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" (UID: "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:55.270167 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.270002 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:55.270236 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.270180 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:55.271682 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.271652 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-dshm" pod "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" (UID: "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:59:55.271802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.271716 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-kube-api-access-zcknw" pod "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" (UID: "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf"). InnerVolumeSpecName "kube-api-access-zcknw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:59:55.271802 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.271785 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tls-certs" pod "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" (UID: "3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:59:55.371211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.371169 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:55.371211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.371209 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:55.371211 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.371218 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcknw\" (UniqueName: \"kubernetes.io/projected/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-kube-api-access-zcknw\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:55.371546 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.371230 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 15:59:55.544385 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.544353 4401 generic.go:356] "Generic (PLEG): container finished" podID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" containerID="742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f" exitCode=0 Oct 30 15:59:55.544549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.544399 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" Oct 30 15:59:55.544549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.544445 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" event={"ID":"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf","Type":"ContainerDied","Data":"742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f"} Oct 30 15:59:55.544549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.544491 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2" event={"ID":"3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf","Type":"ContainerDied","Data":"e3a1344a9601bd85d19367efa520fb33f2158d56f27ce392e64d0beabac5d2e4"} Oct 30 15:59:55.544549 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.544514 4401 scope.go:122] "RemoveContainer" containerID="742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f" Oct 30 15:59:55.555102 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.555082 4401 scope.go:122] "RemoveContainer" containerID="742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f" Oct 30 15:59:55.555431 ip-10-0-133-222 kubenswrapper[4401]: E1030 15:59:55.555409 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f\": container with ID starting with 742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f not found: ID does not exist" containerID="742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f" Oct 30 15:59:55.555489 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.555443 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f"} err="failed to get container status \"742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f\": rpc error: code = NotFound desc = could not find container \"742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f\": container with ID starting with 742eb7307390870b4d8387036dbedadff93d65f13dd0652dac4344c8609f4f7f not found: ID does not exist" Oct 30 15:59:55.604282 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.604247 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2"] Oct 30 15:59:55.608633 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:55.608599 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ba9db9/distinct-2-w5-ba9db9-kserve-7ffdf9f598-lbwz2"] Oct 30 15:59:57.427412 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:57.427377 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" path="/var/lib/kubelet/pods/3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf/volumes" Oct 30 15:59:59.108923 ip-10-0-133-222 kubenswrapper[4401]: I1030 15:59:59.108878 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" podUID="72b2a776-bd23-41f3-908d-81cfc68ffb47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:00:00.042455 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.042414 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749"] Oct 30 16:00:00.057107 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.057056 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c749da43-z9749"] Oct 30 16:00:00.061511 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.061460 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2"] Oct 30 16:00:00.065973 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.065935 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-820c1119-d6sn2"] Oct 30 16:00:00.117597 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.117556 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" podUID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:00:00.166309 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.166269 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl"] Oct 30 16:00:00.166755 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.166740 4401 cpu_manager.go:414] "containerMap: removing container" podUID="28f6e172-6f85-4f93-ba69-f9cf055dc901" containerName="revoke-keys" Oct 30 16:00:00.166797 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.166755 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28f6e172-6f85-4f93-ba69-f9cf055dc901" containerName="revoke-keys" Oct 30 16:00:00.166797 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.166770 4401 cpu_manager.go:414] "containerMap: removing container" podUID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" containerName="main" Oct 30 16:00:00.166797 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.166776 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" containerName="main" Oct 30 16:00:00.166911 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.166869 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="28f6e172-6f85-4f93-ba69-f9cf055dc901" containerName="revoke-keys" Oct 30 16:00:00.166911 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.166879 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc8d8c2-582c-4cc4-b9b1-bfad6501a2bf" containerName="main" Oct 30 16:00:00.171859 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.171817 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.188906 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.188872 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:00:00.189105 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.188956 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:00:00.189707 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.189681 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:00:00.193761 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.193727 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl"] Oct 30 16:00:00.299818 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.299723 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:00:00.322682 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.322641 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.322899 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.322704 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r224q\" (UniqueName: \"kubernetes.io/projected/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-kube-api-access-r224q\") pod \"maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.423496 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.423456 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.423677 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.423520 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r224q\" (UniqueName: \"kubernetes.io/projected/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-kube-api-access-r224q\") pod \"maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.424156 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.424131 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.450331 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.450301 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r224q\" (UniqueName: \"kubernetes.io/projected/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-kube-api-access-r224q\") pod \"maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.487741 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.487703 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:00.806301 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:00.806263 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl"] Oct 30 16:00:00.811263 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:00:00.811233 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f0a1d2_1bb4_4eb1_ba1d_38b8f1d5be79.slice/crio-55f692ae2573b0a499e495497553827db81a7780b4dba6ba3e4bbb8c559ea1f7 WatchSource:0}: Error finding container 55f692ae2573b0a499e495497553827db81a7780b4dba6ba3e4bbb8c559ea1f7: Status 404 returned error can't find the container with id 55f692ae2573b0a499e495497553827db81a7780b4dba6ba3e4bbb8c559ea1f7 Oct 30 16:00:01.443756 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:01.443713 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e58ae06-a500-48ab-9882-e1185ab344e2" path="/var/lib/kubelet/pods/5e58ae06-a500-48ab-9882-e1185ab344e2/volumes" Oct 30 16:00:01.445019 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:01.444991 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6aa618c-d769-4e40-bd45-4ef47de32ae7" path="/var/lib/kubelet/pods/a6aa618c-d769-4e40-bd45-4ef47de32ae7/volumes" Oct 30 16:00:01.633536 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:01.633494 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" event={"ID":"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79","Type":"ContainerStarted","Data":"9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481"} Oct 30 16:00:01.633856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:01.633811 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" event={"ID":"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79","Type":"ContainerStarted","Data":"55f692ae2573b0a499e495497553827db81a7780b4dba6ba3e4bbb8c559ea1f7"} Oct 30 16:00:01.710732 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:01.710594 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" podStartSLOduration=1.71055596 podStartE2EDuration="1.71055596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:01.694697878 +0000 UTC m=+1310.393971037" watchObservedRunningTime="2026-10-30 16:00:01.71055596 +0000 UTC m=+1310.409829118" Oct 30 16:00:02.355907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:02.350906 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl"] Oct 30 16:00:02.659303 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:02.659207 4401 generic.go:356] "Generic (PLEG): container finished" podID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerID="9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481" exitCode=60 Oct 30 16:00:02.660724 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:02.660090 4401 scope.go:122] "RemoveContainer" containerID="9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481" Oct 30 16:00:02.661338 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:02.660398 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" event={"ID":"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79","Type":"ContainerDied","Data":"9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481"} Oct 30 16:00:03.024093 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:03.020487 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb"] Oct 30 16:00:03.024807 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:03.024767 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba9db9-622xb"] Oct 30 16:00:03.427775 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:03.427736 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c93106-f3e9-4461-ae1a-e64e99e282b7" path="/var/lib/kubelet/pods/c6c93106-f3e9-4461-ae1a-e64e99e282b7/volumes" Oct 30 16:00:03.672200 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:03.672156 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" event={"ID":"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79","Type":"ContainerStarted","Data":"2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d"} Oct 30 16:00:03.672200 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:03.672188 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerName="cleanup" containerID="cri-o://2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d" gracePeriod=30 Oct 30 16:00:09.108881 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:09.108836 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" podUID="72b2a776-bd23-41f3-908d-81cfc68ffb47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:00:10.117527 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.117487 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" podUID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:00:10.158779 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.158742 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g"] Oct 30 16:00:10.159018 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.158979 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" containerID="cri-o://74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d" gracePeriod=30 Oct 30 16:00:10.172381 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.172344 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c"] Oct 30 16:00:10.172599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.172563 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" containerID="cri-o://1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121" gracePeriod=30 Oct 30 16:00:10.181177 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.181138 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd"] Oct 30 16:00:10.181400 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.181375 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" containerID="cri-o://ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673" gracePeriod=30 Oct 30 16:00:10.185784 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.185753 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m"] Oct 30 16:00:10.185988 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.185967 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" containerID="cri-o://c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0" gracePeriod=30 Oct 30 16:00:10.908670 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.908640 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 16:00:10.939444 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:10.939413 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 16:00:11.009978 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.009945 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b2a776-bd23-41f3-908d-81cfc68ffb47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2a776-bd23-41f3-908d-81cfc68ffb47-plugins-config-volume\") pod \"72b2a776-bd23-41f3-908d-81cfc68ffb47\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " Oct 30 16:00:11.010182 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.009994 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-plugins-config-volume\") pod \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " Oct 30 16:00:11.010182 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.010045 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b2a776-bd23-41f3-908d-81cfc68ffb47-kube-api-access-zkfb6\" (UniqueName: \"kubernetes.io/projected/72b2a776-bd23-41f3-908d-81cfc68ffb47-kube-api-access-zkfb6\") pod \"72b2a776-bd23-41f3-908d-81cfc68ffb47\" (UID: \"72b2a776-bd23-41f3-908d-81cfc68ffb47\") " Oct 30 16:00:11.010182 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.010129 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-kube-api-access-qgjds\" (UniqueName: \"kubernetes.io/projected/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-kube-api-access-qgjds\") pod \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\" (UID: \"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407\") " Oct 30 16:00:11.010531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.010497 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-plugins-config-volume" pod "7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" (UID: "7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:11.010917 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.010884 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b2a776-bd23-41f3-908d-81cfc68ffb47-plugins-config-volume" pod "72b2a776-bd23-41f3-908d-81cfc68ffb47" (UID: "72b2a776-bd23-41f3-908d-81cfc68ffb47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:11.012481 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.012447 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-kube-api-access-qgjds" pod "7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" (UID: "7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407"). InnerVolumeSpecName "kube-api-access-qgjds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:11.012576 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.012478 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b2a776-bd23-41f3-908d-81cfc68ffb47-kube-api-access-zkfb6" pod "72b2a776-bd23-41f3-908d-81cfc68ffb47" (UID: "72b2a776-bd23-41f3-908d-81cfc68ffb47"). InnerVolumeSpecName "kube-api-access-zkfb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:11.111317 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.111278 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:11.111317 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.111313 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfb6\" (UniqueName: \"kubernetes.io/projected/72b2a776-bd23-41f3-908d-81cfc68ffb47-kube-api-access-zkfb6\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:11.111317 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.111323 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgjds\" (UniqueName: \"kubernetes.io/projected/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407-kube-api-access-qgjds\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:11.111552 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.111332 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72b2a776-bd23-41f3-908d-81cfc68ffb47-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:11.753933 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.753809 4401 generic.go:356] "Generic (PLEG): container finished" podID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" containerID="5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e" exitCode=0 Oct 30 16:00:11.753933 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.753891 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" Oct 30 16:00:11.753933 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.753895 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" event={"ID":"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407","Type":"ContainerDied","Data":"5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e"} Oct 30 16:00:11.754490 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.753982 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9" event={"ID":"7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407","Type":"ContainerDied","Data":"df383edff83c64c291f75fe6158c70b49c4a041d9f6abc200794bfe1ee0c3ceb"} Oct 30 16:00:11.754490 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.754009 4401 scope.go:122] "RemoveContainer" containerID="5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e" Oct 30 16:00:11.755892 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.755864 4401 generic.go:356] "Generic (PLEG): container finished" podID="72b2a776-bd23-41f3-908d-81cfc68ffb47" containerID="8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c" exitCode=0 Oct 30 16:00:11.756117 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.755911 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" event={"ID":"72b2a776-bd23-41f3-908d-81cfc68ffb47","Type":"ContainerDied","Data":"8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c"} Oct 30 16:00:11.756187 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.756161 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" event={"ID":"72b2a776-bd23-41f3-908d-81cfc68ffb47","Type":"ContainerDied","Data":"13dc7dc5d559c69939072b267344cfe6cf58fd70dbe501501498d25fdd4d7926"} Oct 30 16:00:11.756250 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.755989 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h" Oct 30 16:00:11.764975 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.764952 4401 scope.go:122] "RemoveContainer" containerID="5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e" Oct 30 16:00:11.765311 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:11.765289 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e\": container with ID starting with 5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e not found: ID does not exist" containerID="5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e" Oct 30 16:00:11.765378 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.765320 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e"} err="failed to get container status \"5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e\": rpc error: code = NotFound desc = could not find container \"5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e\": container with ID starting with 5b6412fe814da5e915c7f7916c012726f0bdea3ac9c6f52eec1714dd7e1c8a1e not found: ID does not exist" Oct 30 16:00:11.765378 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.765339 4401 scope.go:122] "RemoveContainer" containerID="8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c" Oct 30 16:00:11.776126 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.776099 4401 scope.go:122] "RemoveContainer" containerID="8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c" Oct 30 16:00:11.776452 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:11.776427 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c\": container with ID starting with 8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c not found: ID does not exist" containerID="8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c" Oct 30 16:00:11.776556 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.776457 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c"} err="failed to get container status \"8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c\": rpc error: code = NotFound desc = could not find container \"8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c\": container with ID starting with 8767dbd96d93ed324e2113f9eeabe12f873dba55af8160e17a05252ccd27df2c not found: ID does not exist" Oct 30 16:00:11.817906 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.817858 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9"] Oct 30 16:00:11.830274 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.830235 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba9db9-68894488f9-f4wn9"] Oct 30 16:00:11.870897 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.870856 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h"] Oct 30 16:00:11.876954 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:11.876919 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba9db9-74f5d8664b-rdj2h"] Oct 30 16:00:12.144577 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.144541 4401 scope.go:122] "RemoveContainer" containerID="ab326fe6ee9f9ab38a8e2f84745d0b69d173f83b8de9683cec68f8f09821f259" Oct 30 16:00:12.155644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.155622 4401 scope.go:122] "RemoveContainer" containerID="af69c054864ef9553a3d4b7d5065a7974fb57d80fa564d3bb04069b5d1e795ef" Oct 30 16:00:12.166026 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.166000 4401 scope.go:122] "RemoveContainer" containerID="a290eecbab58d210cf975654525d2864bba23ba214755152ae178eab50ea5ee3" Oct 30 16:00:12.176075 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.176051 4401 scope.go:122] "RemoveContainer" containerID="9b54bbca16a935dee95d9c96a8e072ee32bae1e62b84a2493554f06d7dadb368" Oct 30 16:00:12.186597 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.186568 4401 scope.go:122] "RemoveContainer" containerID="4776eb456a670d60d34cec27848522b18f626cb86ab6fffce60419c4665e0ed4" Oct 30 16:00:12.191937 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.191911 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:00:12.197650 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.197625 4401 scope.go:122] "RemoveContainer" containerID="fcf6c9466238f4084fb55922d1229bf5bdacfcec8fe430b08e8f60ea9c696109" Oct 30 16:00:12.531257 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.531175 4401 patch_prober.go:29] interesting pod/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:00:12.531257 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:12.531235 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:00:13.427715 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:13.427672 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b2a776-bd23-41f3-908d-81cfc68ffb47" path="/var/lib/kubelet/pods/72b2a776-bd23-41f3-908d-81cfc68ffb47/volumes" Oct 30 16:00:13.428267 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:13.428245 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" path="/var/lib/kubelet/pods/7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407/volumes" Oct 30 16:00:15.503858 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.503802 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 16:00:15.511208 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.511175 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 16:00:15.550357 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.550328 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 16:00:15.562496 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562465 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a807894c-cb95-462f-92da-61cb4a059e95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a807894c-cb95-462f-92da-61cb4a059e95-istiod-ca-cert\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.562667 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562512 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-kube-api-access-jwnq9\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-kube-api-access-jwnq9\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.562667 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562581 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-istio-token\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-istio-token\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.562667 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562615 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-certs\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.562667 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562643 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-envoy\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.562943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562683 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-credential-socket\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.562943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562723 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-token\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-token\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.562943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562760 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-socket\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.562943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562803 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istiod-ca-cert\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.562943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562899 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-data\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.562943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562931 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a807894c-cb95-462f-92da-61cb4a059e95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a807894c-cb95-462f-92da-61cb4a059e95-istio-podinfo\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563007 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-socket\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563042 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-kube-api-access-v2zdj\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-kube-api-access-v2zdj\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.562897 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a807894c-cb95-462f-92da-61cb4a059e95-istiod-ca-cert" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563076 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-data\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563090 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-credential-socket" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563114 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-podinfo\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563148 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-certs\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563149 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-certs" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563178 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-envoy\") pod \"a807894c-cb95-462f-92da-61cb4a059e95\" (UID: \"a807894c-cb95-462f-92da-61cb4a059e95\") " Oct 30 16:00:15.563276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563217 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-credential-socket\") pod \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\" (UID: \"2bfdc81f-a987-4bad-903d-ece8c84f97e5\") " Oct 30 16:00:15.563800 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563560 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istiod-ca-cert" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:15.563800 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563646 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 16:00:15.563800 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563779 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-credential-socket" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.563994 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.563812 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-data" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.564054 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564010 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-data" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.564342 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564313 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.564342 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564344 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.564531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564362 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.564531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564377 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.564531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564390 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.564531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564403 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.564531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564416 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a807894c-cb95-462f-92da-61cb4a059e95-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.564897 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564638 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-socket" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.564897 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.564845 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-socket" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.565059 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.565037 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-certs" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.566336 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.566304 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-podinfo" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:00:15.566620 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.566599 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-kube-api-access-v2zdj" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "kube-api-access-v2zdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.566743 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.566718 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-envoy" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.566931 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.566900 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-token" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.568287 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.567891 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a807894c-cb95-462f-92da-61cb4a059e95-istio-podinfo" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:00:15.568721 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.568691 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-kube-api-access-jwnq9" pod "2bfdc81f-a987-4bad-903d-ece8c84f97e5" (UID: "2bfdc81f-a987-4bad-903d-ece8c84f97e5"). InnerVolumeSpecName "kube-api-access-jwnq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.568908 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.568809 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-envoy" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.571749 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.571704 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-istio-token" pod "a807894c-cb95-462f-92da-61cb4a059e95" (UID: "a807894c-cb95-462f-92da-61cb4a059e95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.665531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665492 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-data\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.665531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665531 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-certs\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.665883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665573 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-data\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.665883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665595 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f8600c-0b67-455c-aae8-d506e529300b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f8600c-0b67-455c-aae8-d506e529300b-istiod-ca-cert\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.665883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665640 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-credential-socket\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.665883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665716 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-podinfo\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.665883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665754 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-kube-api-access-rt8jn\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-kube-api-access-rt8jn\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.665883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665797 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f8600c-0b67-455c-aae8-d506e529300b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f8600c-0b67-455c-aae8-d506e529300b-istio-podinfo\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665848 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-certs" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665910 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-data" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.665873 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-istio-token\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-istio-token\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666057 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-socket\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666062 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-credential-socket" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666091 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-kube-api-access-qr2j5\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-kube-api-access-qr2j5\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666114 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-data" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666124 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f8600c-0b67-455c-aae8-d506e529300b-istiod-ca-cert" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666150 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istiod-ca-cert\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666186 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-credential-socket\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.666238 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666221 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-socket\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.666815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666262 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-certs\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.666815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666290 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-envoy\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.666815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666316 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-socket" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.666815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666322 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-envoy\") pod \"27f8600c-0b67-455c-aae8-d506e529300b\" (UID: \"27f8600c-0b67-455c-aae8-d506e529300b\") " Oct 30 16:00:15.666815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666391 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-token\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-token\") pod \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\" (UID: \"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3\") " Oct 30 16:00:15.667115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.666968 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-socket" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.667115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667020 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667047 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2zdj\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-kube-api-access-v2zdj\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667064 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667078 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-certs" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.667115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667097 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667115 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a807894c-cb95-462f-92da-61cb4a059e95-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667129 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667143 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnq9\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-kube-api-access-jwnq9\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667177 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a807894c-cb95-462f-92da-61cb4a059e95-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667195 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667209 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667222 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667253 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bfdc81f-a987-4bad-903d-ece8c84f97e5-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667268 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bfdc81f-a987-4bad-903d-ece8c84f97e5-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667283 4401 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-data\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667300 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f8600c-0b67-455c-aae8-d506e529300b-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667312 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a807894c-cb95-462f-92da-61cb4a059e95-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.667478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667342 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.668092 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.667883 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-credential-socket" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.668428 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.668399 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istiod-ca-cert" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:15.668925 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.668884 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-kube-api-access-qr2j5" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "kube-api-access-qr2j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.669022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.668936 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-kube-api-access-rt8jn" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "kube-api-access-rt8jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.669224 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.669205 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-podinfo" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:00:15.670345 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.670314 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-envoy" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.670649 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.670618 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-istio-token" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.670736 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.670676 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f8600c-0b67-455c-aae8-d506e529300b-istio-podinfo" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:00:15.670775 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.670737 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-token" pod "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" (UID: "e3df45a1-fa98-4c66-82a1-f17e2d1f42f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:15.671357 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.671335 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-envoy" pod "27f8600c-0b67-455c-aae8-d506e529300b" (UID: "27f8600c-0b67-455c-aae8-d506e529300b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768652 4401 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istiod-ca-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768687 4401 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-credential-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768700 4401 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-socket\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768712 4401 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-workload-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768724 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768735 4401 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f8600c-0b67-455c-aae8-d506e529300b-istio-envoy\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768746 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.768751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768757 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.769141 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768770 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt8jn\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-kube-api-access-rt8jn\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.769141 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768783 4401 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f8600c-0b67-455c-aae8-d506e529300b-istio-podinfo\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.769141 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768794 4401 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f8600c-0b67-455c-aae8-d506e529300b-istio-token\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.769141 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.768805 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr2j5\" (UniqueName: \"kubernetes.io/projected/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3-kube-api-access-qr2j5\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:15.787682 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.787646 4401 generic.go:356] "Generic (PLEG): container finished" podID="a807894c-cb95-462f-92da-61cb4a059e95" containerID="1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121" exitCode=0 Oct 30 16:00:15.787901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.787697 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" Oct 30 16:00:15.787901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.787729 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" event={"ID":"a807894c-cb95-462f-92da-61cb4a059e95","Type":"ContainerDied","Data":"1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121"} Oct 30 16:00:15.787901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.787784 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c" event={"ID":"a807894c-cb95-462f-92da-61cb4a059e95","Type":"ContainerDied","Data":"5f0c6fb3d6149c8eaa4e08dc7e8dd3be365e49859601ad32b6e21285a34b3239"} Oct 30 16:00:15.787901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.787807 4401 scope.go:122] "RemoveContainer" containerID="1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121" Oct 30 16:00:15.789243 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.789219 4401 generic.go:356] "Generic (PLEG): container finished" podID="27f8600c-0b67-455c-aae8-d506e529300b" containerID="c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0" exitCode=0 Oct 30 16:00:15.789339 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.789270 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" Oct 30 16:00:15.789339 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.789300 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" event={"ID":"27f8600c-0b67-455c-aae8-d506e529300b","Type":"ContainerDied","Data":"c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0"} Oct 30 16:00:15.789453 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.789344 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m" event={"ID":"27f8600c-0b67-455c-aae8-d506e529300b","Type":"ContainerDied","Data":"fffb2d66d2901353528aa3af61cd67bb3b304d86634488dc3eeb7760da863e06"} Oct 30 16:00:15.791131 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.791099 4401 generic.go:356] "Generic (PLEG): container finished" podID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerID="ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673" exitCode=0 Oct 30 16:00:15.791266 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.791140 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" Oct 30 16:00:15.791266 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.791187 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" event={"ID":"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3","Type":"ContainerDied","Data":"ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673"} Oct 30 16:00:15.791266 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.791230 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd" event={"ID":"e3df45a1-fa98-4c66-82a1-f17e2d1f42f3","Type":"ContainerDied","Data":"e8cea90f35bb2dc93f35ea841aa53dfe3acd8196d6e2c3a80346b8daad7dace7"} Oct 30 16:00:15.796615 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.796576 4401 generic.go:356] "Generic (PLEG): container finished" podID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerID="74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d" exitCode=0 Oct 30 16:00:15.796773 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.796733 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" event={"ID":"2bfdc81f-a987-4bad-903d-ece8c84f97e5","Type":"ContainerDied","Data":"74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d"} Oct 30 16:00:15.796868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.796851 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" event={"ID":"2bfdc81f-a987-4bad-903d-ece8c84f97e5","Type":"ContainerDied","Data":"ba953056828362f6d99c6b7ba1058326046201015460ba44c18475b4fac2d7ce"} Oct 30 16:00:15.796990 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.796975 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g" Oct 30 16:00:15.808889 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.808856 4401 scope.go:122] "RemoveContainer" containerID="1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121" Oct 30 16:00:15.809366 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:15.809338 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121\": container with ID starting with 1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121 not found: ID does not exist" containerID="1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121" Oct 30 16:00:15.809472 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.809373 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121"} err="failed to get container status \"1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121\": rpc error: code = NotFound desc = could not find container \"1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121\": container with ID starting with 1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121 not found: ID does not exist" Oct 30 16:00:15.809472 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.809396 4401 scope.go:122] "RemoveContainer" containerID="c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0" Oct 30 16:00:15.820249 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.820225 4401 scope.go:122] "RemoveContainer" containerID="c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0" Oct 30 16:00:15.820535 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:15.820514 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0\": container with ID starting with c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0 not found: ID does not exist" containerID="c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0" Oct 30 16:00:15.820582 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.820544 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0"} err="failed to get container status \"c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0\": rpc error: code = NotFound desc = could not find container \"c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0\": container with ID starting with c2039944ffba4a5df6a7a89638dff02a47dc44feeb4d31da4cf67b73bf066af0 not found: ID does not exist" Oct 30 16:00:15.820582 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.820562 4401 scope.go:122] "RemoveContainer" containerID="ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673" Oct 30 16:00:15.830579 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.830557 4401 scope.go:122] "RemoveContainer" containerID="ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673" Oct 30 16:00:15.830878 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:15.830854 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673\": container with ID starting with ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673 not found: ID does not exist" containerID="ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673" Oct 30 16:00:15.830960 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.830890 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673"} err="failed to get container status \"ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673\": rpc error: code = NotFound desc = could not find container \"ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673\": container with ID starting with ca7696dc834e07a63de8ba1906f989ff4b962d2fb2de4152452dd7834859d673 not found: ID does not exist" Oct 30 16:00:15.830960 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.830914 4401 scope.go:122] "RemoveContainer" containerID="74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d" Oct 30 16:00:15.840022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.840002 4401 scope.go:122] "RemoveContainer" containerID="74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d" Oct 30 16:00:15.840304 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:15.840281 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d\": container with ID starting with 74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d not found: ID does not exist" containerID="74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d" Oct 30 16:00:15.840375 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.840316 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d"} err="failed to get container status \"74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d\": rpc error: code = NotFound desc = could not find container \"74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d\": container with ID starting with 74509af2b34aa04a83f68b5debacc4b752464f696523a3151a285c0b9592975d not found: ID does not exist" Oct 30 16:00:15.901762 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.901719 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m"] Oct 30 16:00:15.920740 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.920701 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-n9j8m"] Oct 30 16:00:15.989467 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:15.989425 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g"] Oct 30 16:00:16.034117 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:16.034019 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-hj76g"] Oct 30 16:00:16.075682 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:16.075642 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x"] Oct 30 16:00:16.095540 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:16.095504 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4080bf2-2864x"] Oct 30 16:00:16.100405 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:16.100379 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd"] Oct 30 16:00:16.104841 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:16.104792 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-bv7qd"] Oct 30 16:00:16.155629 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:16.155586 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c"] Oct 30 16:00:16.180613 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:16.180573 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba9db9-openshift-default-8b9b5cf77-khg8c"] Oct 30 16:00:17.427494 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:17.427461 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f8600c-0b67-455c-aae8-d506e529300b" path="/var/lib/kubelet/pods/27f8600c-0b67-455c-aae8-d506e529300b/volumes" Oct 30 16:00:17.428004 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:17.427912 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" path="/var/lib/kubelet/pods/2bfdc81f-a987-4bad-903d-ece8c84f97e5/volumes" Oct 30 16:00:17.428320 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:17.428306 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a807894c-cb95-462f-92da-61cb4a059e95" path="/var/lib/kubelet/pods/a807894c-cb95-462f-92da-61cb4a059e95/volumes" Oct 30 16:00:17.429064 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:17.429048 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" path="/var/lib/kubelet/pods/e3df45a1-fa98-4c66-82a1-f17e2d1f42f3/volumes" Oct 30 16:00:17.429436 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:17.429423 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ea5671-2313-4732-9017-35d41dd17635" path="/var/lib/kubelet/pods/e8ea5671-2313-4732-9017-35d41dd17635/volumes" Oct 30 16:00:18.469509 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.469469 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5964675dcd-5db52"] Oct 30 16:00:18.469972 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.469955 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" Oct 30 16:00:18.470027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.469974 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" Oct 30 16:00:18.470027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.469987 4401 cpu_manager.go:414] "containerMap: removing container" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" Oct 30 16:00:18.470027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.469993 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" Oct 30 16:00:18.470027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470000 4401 cpu_manager.go:414] "containerMap: removing container" podUID="72b2a776-bd23-41f3-908d-81cfc68ffb47" containerName="payload-processing" Oct 30 16:00:18.470027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470006 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b2a776-bd23-41f3-908d-81cfc68ffb47" containerName="payload-processing" Oct 30 16:00:18.470027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470019 4401 cpu_manager.go:414] "containerMap: removing container" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" Oct 30 16:00:18.470027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470024 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470037 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" containerName="payload-pre-processing" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470042 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" containerName="payload-pre-processing" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470058 4401 cpu_manager.go:414] "containerMap: removing container" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470062 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470125 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="a807894c-cb95-462f-92da-61cb4a059e95" containerName="istio-proxy" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470137 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3df45a1-fa98-4c66-82a1-f17e2d1f42f3" containerName="istio-proxy" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470144 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bfdc81f-a987-4bad-903d-ece8c84f97e5" containerName="istio-proxy" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470150 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b2a776-bd23-41f3-908d-81cfc68ffb47" containerName="payload-processing" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470157 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f8600c-0b67-455c-aae8-d506e529300b" containerName="istio-proxy" Oct 30 16:00:18.470248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.470164 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d911f8e-cab9-4f3d-a1f1-c2a08cd2f407" containerName="payload-pre-processing" Oct 30 16:00:18.475675 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.475647 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.497246 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.497206 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 16:00:18.497246 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.497209 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lcl2r\"" Oct 30 16:00:18.519016 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.518970 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5964675dcd-5db52"] Oct 30 16:00:18.519431 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.519403 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 16:00:18.594753 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.594706 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxzz\" (UniqueName: \"kubernetes.io/projected/3998a246-ed07-41ae-aa41-e2c9f1b2172c-kube-api-access-8pxzz\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.595010 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.594877 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-webhook-cert\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.595010 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.594950 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-metrics-certs\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.599491 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.599467 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 16:00:18.696397 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.696354 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxzz\" (UniqueName: \"kubernetes.io/projected/3998a246-ed07-41ae-aa41-e2c9f1b2172c-kube-api-access-8pxzz\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.696582 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.696445 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-webhook-cert\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.696582 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.696478 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-metrics-certs\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.699153 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.699119 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-webhook-cert\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.699293 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.699177 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-metrics-certs\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.752455 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.752368 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxzz\" (UniqueName: \"kubernetes.io/projected/3998a246-ed07-41ae-aa41-e2c9f1b2172c-kube-api-access-8pxzz\") pod \"maas-controller-5964675dcd-5db52\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:18.795197 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:18.795154 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:19.050793 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:19.050705 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc"] Oct 30 16:00:19.060070 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:19.060039 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-107f9199-4wmxc"] Oct 30 16:00:19.138228 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:19.138205 4401 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:00:19.138613 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:19.138593 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5964675dcd-5db52"] Oct 30 16:00:19.426991 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:19.426943 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d17ed20-b1c9-419b-b8b4-989fc79fc597" path="/var/lib/kubelet/pods/8d17ed20-b1c9-419b-b8b4-989fc79fc597/volumes" Oct 30 16:00:19.838753 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:19.838670 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5964675dcd-5db52" event={"ID":"3998a246-ed07-41ae-aa41-e2c9f1b2172c","Type":"ContainerStarted","Data":"1a624837b5349d852630c5bb96d0e694f383e46da489e56f814a66adee81d7ba"} Oct 30 16:00:21.858656 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:21.858615 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5964675dcd-5db52" event={"ID":"3998a246-ed07-41ae-aa41-e2c9f1b2172c","Type":"ContainerStarted","Data":"c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee"} Oct 30 16:00:21.859092 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:21.858758 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:21.916437 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:21.916374 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5964675dcd-5db52" podStartSLOduration=1.567910721 podStartE2EDuration="3.916357816s" totalImagesPullingTime="2.348447095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:21.912084026 +0000 UTC m=+1330.611357183" watchObservedRunningTime="2026-10-30 16:00:21.916357816 +0000 UTC m=+1330.615630972" Oct 30 16:00:21.967953 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:21.967915 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:00:22.034758 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:22.034718 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2"] Oct 30 16:00:22.038651 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:22.038617 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1de807e0-876l2"] Oct 30 16:00:23.427055 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:23.427019 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7847a62-f8dd-43ce-b5cb-53ba70ba5fae" path="/var/lib/kubelet/pods/b7847a62-f8dd-43ce-b5cb-53ba70ba5fae/volumes" Oct 30 16:00:32.034728 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:32.034691 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8"] Oct 30 16:00:32.048709 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:32.048676 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c3e6ae0-lmmj8"] Oct 30 16:00:32.868720 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:32.868685 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:33.427629 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.427596 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c07d02-b5ef-49cf-8c4e-bc693da6ef65" path="/var/lib/kubelet/pods/e3c07d02-b5ef-49cf-8c4e-bc693da6ef65/volumes" Oct 30 16:00:33.743717 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:33.743654 4401 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3df45a1_fa98_4c66_82a1_f17e2d1f42f3.slice/crio-e8cea90f35bb2dc93f35ea841aa53dfe3acd8196d6e2c3a80346b8daad7dace7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfdc81f_a987_4bad_903d_ece8c84f97e5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f8600c_0b67_455c_aae8_d506e529300b.slice/crio-fffb2d66d2901353528aa3af61cd67bb3b304d86634488dc3eeb7760da863e06\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f8600c_0b67_455c_aae8_d506e529300b.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfdc81f_a987_4bad_903d_ece8c84f97e5.slice/crio-ba953056828362f6d99c6b7ba1058326046201015460ba44c18475b4fac2d7ce\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3df45a1_fa98_4c66_82a1_f17e2d1f42f3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda807894c_cb95_462f_92da_61cb4a059e95.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda807894c_cb95_462f_92da_61cb4a059e95.slice/crio-1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda807894c_cb95_462f_92da_61cb4a059e95.slice/crio-5f0c6fb3d6149c8eaa4e08dc7e8dd3be365e49859601ad32b6e21285a34b3239\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f0a1d2_1bb4_4eb1_ba1d_38b8f1d5be79.slice/crio-2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f0a1d2_1bb4_4eb1_ba1d_38b8f1d5be79.slice/crio-conmon-2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda807894c_cb95_462f_92da_61cb4a059e95.slice/crio-conmon-1cc387caa59bbdda65b9fa9c16a422aeb91044708dd1cf8400e4cdb29d94b121.scope\": RecentStats: unable to find data in memory cache]" Oct 30 16:00:33.888313 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.888278 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:33.955322 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.955222 4401 generic.go:356] "Generic (PLEG): container finished" podID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerID="2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d" exitCode=137 Oct 30 16:00:33.955478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.955341 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" event={"ID":"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79","Type":"ContainerDied","Data":"2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d"} Oct 30 16:00:33.955478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.955377 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" Oct 30 16:00:33.955478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.955439 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl" event={"ID":"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79","Type":"ContainerDied","Data":"55f692ae2573b0a499e495497553827db81a7780b4dba6ba3e4bbb8c559ea1f7"} Oct 30 16:00:33.955478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.955473 4401 scope.go:122] "RemoveContainer" containerID="2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d" Oct 30 16:00:33.968981 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.968957 4401 scope.go:122] "RemoveContainer" containerID="9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481" Oct 30 16:00:33.980448 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.980425 4401 scope.go:122] "RemoveContainer" containerID="2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d" Oct 30 16:00:33.980898 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:33.980868 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d\": container with ID starting with 2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d not found: ID does not exist" containerID="2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d" Oct 30 16:00:33.981044 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.980902 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d"} err="failed to get container status \"2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d\": rpc error: code = NotFound desc = could not find container \"2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d\": container with ID starting with 2006fecfce810c76c2a025e780127e57ba4d75e0dbbfa66330bedf308366e15d not found: ID does not exist" Oct 30 16:00:33.981044 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.980922 4401 scope.go:122] "RemoveContainer" containerID="9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481" Oct 30 16:00:33.981284 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:33.981260 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481\": container with ID starting with 9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481 not found: ID does not exist" containerID="9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481" Oct 30 16:00:33.981333 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:33.981293 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481"} err="failed to get container status \"9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481\": rpc error: code = NotFound desc = could not find container \"9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481\": container with ID starting with 9ae8617dc4d188908fceb09dad9b0511033da472c9900e5e5caa49ec353e7481 not found: ID does not exist" Oct 30 16:00:34.054429 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.054384 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-maas-api-service-ca\") pod \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " Oct 30 16:00:34.054429 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.054436 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-kube-api-access-r224q\" (UniqueName: \"kubernetes.io/projected/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-kube-api-access-r224q\") pod \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\" (UID: \"84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79\") " Oct 30 16:00:34.054778 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.054753 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-maas-api-service-ca" pod "84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" (UID: "84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:34.056499 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.056474 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-kube-api-access-r224q" pod "84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" (UID: "84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79"). InnerVolumeSpecName "kube-api-access-r224q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:34.155793 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.155733 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-maas-api-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:34.155793 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.155784 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r224q\" (UniqueName: \"kubernetes.io/projected/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79-kube-api-access-r224q\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:34.307447 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.307409 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl"] Oct 30 16:00:34.311532 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:34.311499 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ba9db9-29889600-fx6cl"] Oct 30 16:00:35.427361 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:35.427312 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" path="/var/lib/kubelet/pods/84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79/volumes" Oct 30 16:00:39.040442 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.040390 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7"] Oct 30 16:00:39.054103 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.054068 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc695fd-rvkh7"] Oct 30 16:00:39.427032 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.426993 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0a12856-d219-4301-b70c-79c0cbcf4109" path="/var/lib/kubelet/pods/b0a12856-d219-4301-b70c-79c0cbcf4109/volumes" Oct 30 16:00:39.727736 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.727644 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s"] Oct 30 16:00:39.729100 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.729074 4401 cpu_manager.go:414] "containerMap: removing container" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerName="cleanup" Oct 30 16:00:39.729266 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.729252 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerName="cleanup" Oct 30 16:00:39.729372 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.729361 4401 cpu_manager.go:414] "containerMap: removing container" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerName="cleanup" Oct 30 16:00:39.729448 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.729438 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerName="cleanup" Oct 30 16:00:39.729735 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.729720 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerName="cleanup" Oct 30 16:00:39.729857 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.729845 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f0a1d2-1bb4-4eb1-ba1d-38b8f1d5be79" containerName="cleanup" Oct 30 16:00:39.738563 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.738541 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.748234 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.748206 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s"] Oct 30 16:00:39.756527 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.756504 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:00:39.756780 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.756762 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:00:39.769965 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.769936 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:00:39.770142 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.769973 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:00:39.806810 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.806760 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rns9z\" (UniqueName: \"kubernetes.io/projected/55d1dc1b-f796-457a-9d2e-600d90fe51ac-kube-api-access-rns9z\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.807000 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.806920 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.807000 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.806950 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.907658 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.907601 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rns9z\" (UniqueName: \"kubernetes.io/projected/55d1dc1b-f796-457a-9d2e-600d90fe51ac-kube-api-access-rns9z\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.907658 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.907664 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.907976 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.907691 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.908510 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.908484 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.910143 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.910119 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:39.934228 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:39.934183 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rns9z\" (UniqueName: \"kubernetes.io/projected/55d1dc1b-f796-457a-9d2e-600d90fe51ac-kube-api-access-rns9z\") pod \"maas-api-revoke-sub-85237e45-qt79s\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:40.057610 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:40.057518 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:40.374271 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:40.373697 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s"] Oct 30 16:00:40.374426 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:00:40.374337 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d1dc1b_f796_457a_9d2e_600d90fe51ac.slice/crio-8bdc957426e427242e32a1d8acf8765bc8119950296a7e3ab09d2b57def1c163 WatchSource:0}: Error finding container 8bdc957426e427242e32a1d8acf8765bc8119950296a7e3ab09d2b57def1c163: Status 404 returned error can't find the container with id 8bdc957426e427242e32a1d8acf8765bc8119950296a7e3ab09d2b57def1c163 Oct 30 16:00:41.011896 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:41.011776 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" event={"ID":"55d1dc1b-f796-457a-9d2e-600d90fe51ac","Type":"ContainerStarted","Data":"7f4cab3a516d9d4c24d9a7a4fc35d03fe478d7f451dd803af9fd88da8cbde288"} Oct 30 16:00:41.011896 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:41.011849 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" event={"ID":"55d1dc1b-f796-457a-9d2e-600d90fe51ac","Type":"ContainerStarted","Data":"8bdc957426e427242e32a1d8acf8765bc8119950296a7e3ab09d2b57def1c163"} Oct 30 16:00:41.049357 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:41.049314 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j"] Oct 30 16:00:41.057338 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:41.057308 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1669e612-66k4j"] Oct 30 16:00:41.071290 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:41.071239 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" podStartSLOduration=2.071225485 podStartE2EDuration="2.071225485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:41.068697491 +0000 UTC m=+1349.767970649" watchObservedRunningTime="2026-10-30 16:00:41.071225485 +0000 UTC m=+1349.770498642" Oct 30 16:00:41.427273 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:41.427235 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca77415d-2c8c-4c1f-8e1f-8c5642b10a75" path="/var/lib/kubelet/pods/ca77415d-2c8c-4c1f-8e1f-8c5642b10a75/volumes" Oct 30 16:00:42.021183 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:42.021131 4401 generic.go:356] "Generic (PLEG): container finished" podID="55d1dc1b-f796-457a-9d2e-600d90fe51ac" containerID="7f4cab3a516d9d4c24d9a7a4fc35d03fe478d7f451dd803af9fd88da8cbde288" exitCode=0 Oct 30 16:00:42.021371 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:42.021217 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" event={"ID":"55d1dc1b-f796-457a-9d2e-600d90fe51ac","Type":"ContainerDied","Data":"7f4cab3a516d9d4c24d9a7a4fc35d03fe478d7f451dd803af9fd88da8cbde288"} Oct 30 16:00:43.175674 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.175645 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:43.237017 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.236977 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d1dc1b-f796-457a-9d2e-600d90fe51ac-kube-api-access-rns9z\" (UniqueName: \"kubernetes.io/projected/55d1dc1b-f796-457a-9d2e-600d90fe51ac-kube-api-access-rns9z\") pod \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " Oct 30 16:00:43.237205 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.237032 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-openshift-ca\") pod \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " Oct 30 16:00:43.237205 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.237122 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-xks-ca\") pod \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\" (UID: \"55d1dc1b-f796-457a-9d2e-600d90fe51ac\") " Oct 30 16:00:43.237499 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.237472 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-openshift-ca" pod "55d1dc1b-f796-457a-9d2e-600d90fe51ac" (UID: "55d1dc1b-f796-457a-9d2e-600d90fe51ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:00:43.239205 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.239174 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-xks-ca" pod "55d1dc1b-f796-457a-9d2e-600d90fe51ac" (UID: "55d1dc1b-f796-457a-9d2e-600d90fe51ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:00:43.239309 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.239260 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d1dc1b-f796-457a-9d2e-600d90fe51ac-kube-api-access-rns9z" pod "55d1dc1b-f796-457a-9d2e-600d90fe51ac" (UID: "55d1dc1b-f796-457a-9d2e-600d90fe51ac"). InnerVolumeSpecName "kube-api-access-rns9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:43.338184 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.338099 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:43.338184 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.338133 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rns9z\" (UniqueName: \"kubernetes.io/projected/55d1dc1b-f796-457a-9d2e-600d90fe51ac-kube-api-access-rns9z\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:43.338184 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:43.338144 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d1dc1b-f796-457a-9d2e-600d90fe51ac-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:44.037957 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:44.037922 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" Oct 30 16:00:44.038146 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:44.037927 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s" event={"ID":"55d1dc1b-f796-457a-9d2e-600d90fe51ac","Type":"ContainerDied","Data":"8bdc957426e427242e32a1d8acf8765bc8119950296a7e3ab09d2b57def1c163"} Oct 30 16:00:44.038146 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:44.038074 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bdc957426e427242e32a1d8acf8765bc8119950296a7e3ab09d2b57def1c163" Oct 30 16:00:51.616443 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.616399 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5964675dcd-5db52"] Oct 30 16:00:51.617300 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.616626 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5964675dcd-5db52" podUID="3998a246-ed07-41ae-aa41-e2c9f1b2172c" containerName="manager" containerID="cri-o://c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee" gracePeriod=30 Oct 30 16:00:51.870158 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.870091 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:51.926585 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.926522 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-webhook-cert\") pod \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " Oct 30 16:00:51.926787 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.926606 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3998a246-ed07-41ae-aa41-e2c9f1b2172c-kube-api-access-8pxzz\" (UniqueName: \"kubernetes.io/projected/3998a246-ed07-41ae-aa41-e2c9f1b2172c-kube-api-access-8pxzz\") pod \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " Oct 30 16:00:51.926787 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.926737 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-metrics-certs\") pod \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\" (UID: \"3998a246-ed07-41ae-aa41-e2c9f1b2172c\") " Oct 30 16:00:51.929751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.929316 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3998a246-ed07-41ae-aa41-e2c9f1b2172c-kube-api-access-8pxzz" pod "3998a246-ed07-41ae-aa41-e2c9f1b2172c" (UID: "3998a246-ed07-41ae-aa41-e2c9f1b2172c"). InnerVolumeSpecName "kube-api-access-8pxzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:00:51.929751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.929683 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-metrics-certs" pod "3998a246-ed07-41ae-aa41-e2c9f1b2172c" (UID: "3998a246-ed07-41ae-aa41-e2c9f1b2172c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:00:51.930007 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:51.929765 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-webhook-cert" pod "3998a246-ed07-41ae-aa41-e2c9f1b2172c" (UID: "3998a246-ed07-41ae-aa41-e2c9f1b2172c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:00:51.970121 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:51.970077 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:00:52.027818 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.027781 4401 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-webhook-cert\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:52.027818 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.027812 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pxzz\" (UniqueName: \"kubernetes.io/projected/3998a246-ed07-41ae-aa41-e2c9f1b2172c-kube-api-access-8pxzz\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:52.027818 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.027848 4401 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3998a246-ed07-41ae-aa41-e2c9f1b2172c-metrics-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:00:52.100988 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.100954 4401 generic.go:356] "Generic (PLEG): container finished" podID="3998a246-ed07-41ae-aa41-e2c9f1b2172c" containerID="c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee" exitCode=0 Oct 30 16:00:52.101167 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.101003 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5964675dcd-5db52" Oct 30 16:00:52.101167 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.101032 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5964675dcd-5db52" event={"ID":"3998a246-ed07-41ae-aa41-e2c9f1b2172c","Type":"ContainerDied","Data":"c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee"} Oct 30 16:00:52.101167 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.101095 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5964675dcd-5db52" event={"ID":"3998a246-ed07-41ae-aa41-e2c9f1b2172c","Type":"ContainerDied","Data":"1a624837b5349d852630c5bb96d0e694f383e46da489e56f814a66adee81d7ba"} Oct 30 16:00:52.101167 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.101124 4401 scope.go:122] "RemoveContainer" containerID="c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee" Oct 30 16:00:52.112274 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.112253 4401 scope.go:122] "RemoveContainer" containerID="c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee" Oct 30 16:00:52.112603 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:00:52.112584 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee\": container with ID starting with c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee not found: ID does not exist" containerID="c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee" Oct 30 16:00:52.112652 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.112613 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee"} err="failed to get container status \"c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee\": rpc error: code = NotFound desc = could not find container \"c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee\": container with ID starting with c6ab8918f07a9bafa2c36206fe8db8630a58e35a269e808cc2838e0af299d9ee not found: ID does not exist" Oct 30 16:00:52.161623 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.161533 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5964675dcd-5db52"] Oct 30 16:00:52.165293 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:52.165262 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5964675dcd-5db52"] Oct 30 16:00:53.427013 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.426972 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3998a246-ed07-41ae-aa41-e2c9f1b2172c" path="/var/lib/kubelet/pods/3998a246-ed07-41ae-aa41-e2c9f1b2172c/volumes" Oct 30 16:00:53.427546 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.427320 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5964675dcd-kxw7t"] Oct 30 16:00:53.427764 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.427750 4401 cpu_manager.go:414] "containerMap: removing container" podUID="55d1dc1b-f796-457a-9d2e-600d90fe51ac" containerName="revoke-keys" Oct 30 16:00:53.427803 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.427765 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d1dc1b-f796-457a-9d2e-600d90fe51ac" containerName="revoke-keys" Oct 30 16:00:53.427803 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.427794 4401 cpu_manager.go:414] "containerMap: removing container" podUID="3998a246-ed07-41ae-aa41-e2c9f1b2172c" containerName="manager" Oct 30 16:00:53.427803 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.427800 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3998a246-ed07-41ae-aa41-e2c9f1b2172c" containerName="manager" Oct 30 16:00:53.427925 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.427892 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="3998a246-ed07-41ae-aa41-e2c9f1b2172c" containerName="manager" Oct 30 16:00:53.427925 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.427907 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d1dc1b-f796-457a-9d2e-600d90fe51ac" containerName="revoke-keys" Oct 30 16:00:53.433241 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.433218 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.450437 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.450395 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5964675dcd-kxw7t"] Oct 30 16:00:53.459684 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.459649 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 16:00:53.466730 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.466701 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lcl2r\"" Oct 30 16:00:53.466954 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.466934 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 16:00:53.467026 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.466935 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 16:00:53.541041 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.540997 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q89vs\" (UniqueName: \"kubernetes.io/projected/222d4524-f967-4258-bb9c-9252db660785-kube-api-access-q89vs\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.541228 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.541092 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/222d4524-f967-4258-bb9c-9252db660785-webhook-cert\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.541228 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.541195 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222d4524-f967-4258-bb9c-9252db660785-metrics-certs\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.642934 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.642887 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q89vs\" (UniqueName: \"kubernetes.io/projected/222d4524-f967-4258-bb9c-9252db660785-kube-api-access-q89vs\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.643148 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.642945 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/222d4524-f967-4258-bb9c-9252db660785-webhook-cert\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.643148 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.642967 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222d4524-f967-4258-bb9c-9252db660785-metrics-certs\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.645573 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.645547 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/222d4524-f967-4258-bb9c-9252db660785-metrics-certs\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.645716 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.645674 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/222d4524-f967-4258-bb9c-9252db660785-webhook-cert\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.668637 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.668606 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q89vs\" (UniqueName: \"kubernetes.io/projected/222d4524-f967-4258-bb9c-9252db660785-kube-api-access-q89vs\") pod \"maas-controller-5964675dcd-kxw7t\" (UID: \"222d4524-f967-4258-bb9c-9252db660785\") " pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:53.748856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:53.748740 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:54.067619 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:00:54.067579 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222d4524_f967_4258_bb9c_9252db660785.slice/crio-2a733a371719319d7d40efa45b54a845668eba165e773b0b3a26dbb5aa840dca WatchSource:0}: Error finding container 2a733a371719319d7d40efa45b54a845668eba165e773b0b3a26dbb5aa840dca: Status 404 returned error can't find the container with id 2a733a371719319d7d40efa45b54a845668eba165e773b0b3a26dbb5aa840dca Oct 30 16:00:54.068856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.068804 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5964675dcd-kxw7t"] Oct 30 16:00:54.119303 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.119266 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5964675dcd-kxw7t" event={"ID":"222d4524-f967-4258-bb9c-9252db660785","Type":"ContainerStarted","Data":"2a733a371719319d7d40efa45b54a845668eba165e773b0b3a26dbb5aa840dca"} Oct 30 16:00:54.187355 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.187316 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp"] Oct 30 16:00:54.192708 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.192683 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.210887 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.210856 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f"] Oct 30 16:00:54.216883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.214974 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.216883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.215336 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66"] Oct 30 16:00:54.219537 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.219509 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.220796 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.220740 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-x4s56\"" Oct 30 16:00:54.225685 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.225655 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp"] Oct 30 16:00:54.236275 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.236248 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7"] Oct 30 16:00:54.240924 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.240901 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f"] Oct 30 16:00:54.241050 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.241036 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.247897 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.247870 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmsqb\" (UniqueName: \"kubernetes.io/projected/636add60-0495-48b7-bd72-1cebe229fa0c-kube-api-access-rmsqb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248030 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.247919 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248030 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.247992 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248159 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.248079 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248206 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.248183 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248252 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.248203 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636add60-0495-48b7-bd72-1cebe229fa0c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248291 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.248279 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636add60-0495-48b7-bd72-1cebe229fa0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248342 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.248326 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.248379 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.248350 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636add60-0495-48b7-bd72-1cebe229fa0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.269626 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.269590 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66"] Oct 30 16:00:54.295605 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.295567 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7"] Oct 30 16:00:54.348996 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.348963 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636add60-0495-48b7-bd72-1cebe229fa0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.349196 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349010 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.349196 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349029 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.349196 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349047 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.349196 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349153 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636add60-0495-48b7-bd72-1cebe229fa0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.349427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349259 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.349427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349308 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.349427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349338 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd43595-43d2-4f68-85bb-fea131b69e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.349427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349394 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmsqb\" (UniqueName: \"kubernetes.io/projected/636add60-0495-48b7-bd72-1cebe229fa0c-kube-api-access-rmsqb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.349644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349439 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.349644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349477 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.349644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349504 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.349644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349584 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.349644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349596 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.349644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349618 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxggc\" (UniqueName: \"kubernetes.io/projected/6cd43595-43d2-4f68-85bb-fea131b69e72-kube-api-access-pxggc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.349907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349659 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.349907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349685 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.349907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349746 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.349907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349786 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.349907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349851 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.349907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349890 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349855 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/636add60-0495-48b7-bd72-1cebe229fa0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349932 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.349988 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzh7s\" (UniqueName: \"kubernetes.io/projected/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-kube-api-access-dzh7s\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350028 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350057 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350082 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350156 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.350181 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350157 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350206 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6639c2-1eac-4716-8a14-cd33938b1c67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350262 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350281 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350296 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350321 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350417 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350435 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636add60-0495-48b7-bd72-1cebe229fa0c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.350477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350454 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvhmr\" (UniqueName: \"kubernetes.io/projected/8c6639c2-1eac-4716-8a14-cd33938b1c67-kube-api-access-zvhmr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.351024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350484 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.351024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350504 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.351024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350604 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.351024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.350672 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.352058 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.352027 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/636add60-0495-48b7-bd72-1cebe229fa0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.352212 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.352193 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/636add60-0495-48b7-bd72-1cebe229fa0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.374922 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.374884 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/636add60-0495-48b7-bd72-1cebe229fa0c-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.375083 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.374968 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmsqb\" (UniqueName: \"kubernetes.io/projected/636add60-0495-48b7-bd72-1cebe229fa0c-kube-api-access-rmsqb\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-szdqp\" (UID: \"636add60-0495-48b7-bd72-1cebe229fa0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.452215 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452178 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvhmr\" (UniqueName: \"kubernetes.io/projected/8c6639c2-1eac-4716-8a14-cd33938b1c67-kube-api-access-zvhmr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.452215 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452222 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452253 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452391 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452471 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452503 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452535 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452559 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452568 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452612 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd43595-43d2-4f68-85bb-fea131b69e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452637 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452664 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452721 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452752 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.452772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452765 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxggc\" (UniqueName: \"kubernetes.io/projected/6cd43595-43d2-4f68-85bb-fea131b69e72-kube-api-access-pxggc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452841 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452874 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452912 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452947 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.452997 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453029 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzh7s\" (UniqueName: \"kubernetes.io/projected/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-kube-api-access-dzh7s\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453055 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453082 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453109 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453138 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6639c2-1eac-4716-8a14-cd33938b1c67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453190 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453225 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453249 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453262 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453291 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.453515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453465 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cd43595-43d2-4f68-85bb-fea131b69e72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.454373 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453756 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.454373 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.453812 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-workload-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.454373 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.454007 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.454373 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.454094 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.454599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.454476 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.454599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.454511 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6639c2-1eac-4716-8a14-cd33938b1c67-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.454599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.454536 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.454767 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.454729 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-workload-certs\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.454838 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.454775 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-data\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.455045 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.455020 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.455183 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.455077 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-credential-socket\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.455339 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.455319 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.455589 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.455565 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.455847 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.455805 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.455921 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.455900 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.456637 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.456615 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.456737 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.456654 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-envoy\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.478425 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.478390 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6639c2-1eac-4716-8a14-cd33938b1c67-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.478545 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.478399 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvhmr\" (UniqueName: \"kubernetes.io/projected/8c6639c2-1eac-4716-8a14-cd33938b1c67-kube-api-access-zvhmr\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-trg66\" (UID: \"8c6639c2-1eac-4716-8a14-cd33938b1c67\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.478545 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.478534 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cd43595-43d2-4f68-85bb-fea131b69e72-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.478681 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.478584 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-istio-token\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.478743 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.478703 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxggc\" (UniqueName: \"kubernetes.io/projected/6cd43595-43d2-4f68-85bb-fea131b69e72-kube-api-access-pxggc\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-rsp9f\" (UID: \"6cd43595-43d2-4f68-85bb-fea131b69e72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.478920 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.478901 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzh7s\" (UniqueName: \"kubernetes.io/projected/672a1ec4-2d4f-4a6f-8e4a-369b6be336c7-kube-api-access-dzh7s\") pod \"maas-default-gateway-openshift-default-9dccdc9b7-862j7\" (UID: \"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.506511 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.506477 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:54.534009 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.533974 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:54.542948 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.542919 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:54.555329 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.555299 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:54.840972 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:00:54.840940 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod636add60_0495_48b7_bd72_1cebe229fa0c.slice/crio-fe69a768af9ac7167ce6a57ec916de90e90ec48ad2b33e4a692426f5e6db2585 WatchSource:0}: Error finding container fe69a768af9ac7167ce6a57ec916de90e90ec48ad2b33e4a692426f5e6db2585: Status 404 returned error can't find the container with id fe69a768af9ac7167ce6a57ec916de90e90ec48ad2b33e4a692426f5e6db2585 Oct 30 16:00:54.842535 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.842513 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp"] Oct 30 16:00:54.843980 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.843949 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.844039 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.844009 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.844085 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.844040 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.885367 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.885337 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f"] Oct 30 16:00:54.890102 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:00:54.890065 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd43595_43d2_4f68_85bb_fea131b69e72.slice/crio-a0dc227706094548dcf33af96f4fd91948e627a8ec3be0f5af244594b3d5d431 WatchSource:0}: Error finding container a0dc227706094548dcf33af96f4fd91948e627a8ec3be0f5af244594b3d5d431: Status 404 returned error can't find the container with id a0dc227706094548dcf33af96f4fd91948e627a8ec3be0f5af244594b3d5d431 Oct 30 16:00:54.892775 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.892738 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.892912 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.892844 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.892912 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.892875 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.913108 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:00:54.913074 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6639c2_1eac_4716_8a14_cd33938b1c67.slice/crio-54e1a59e7ec4b35f270ef8fa31fe598a1636e8a47620f52f7e16d69e514aee9a WatchSource:0}: Error finding container 54e1a59e7ec4b35f270ef8fa31fe598a1636e8a47620f52f7e16d69e514aee9a: Status 404 returned error can't find the container with id 54e1a59e7ec4b35f270ef8fa31fe598a1636e8a47620f52f7e16d69e514aee9a Oct 30 16:00:54.913968 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.913945 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66"] Oct 30 16:00:54.916153 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.916112 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.916230 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.916206 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.916282 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.916252 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.963564 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.963530 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7"] Oct 30 16:00:54.963680 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:00:54.963539 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672a1ec4_2d4f_4a6f_8e4a_369b6be336c7.slice/crio-8f75bbc23fe18d5eec02fa14bccd8964535c6ebeb72ee3ac079c0a99451077e9 WatchSource:0}: Error finding container 8f75bbc23fe18d5eec02fa14bccd8964535c6ebeb72ee3ac079c0a99451077e9: Status 404 returned error can't find the container with id 8f75bbc23fe18d5eec02fa14bccd8964535c6ebeb72ee3ac079c0a99451077e9 Oct 30 16:00:54.965688 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.965652 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.965816 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.965722 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:54.965816 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:54.965755 4401 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 16:00:55.130631 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.130536 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" event={"ID":"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7","Type":"ContainerStarted","Data":"bb801170a0db52931cfe7d71a1e50b6fa3a6f94a7443b243f3fd70ef9509cc8e"} Oct 30 16:00:55.130897 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.130879 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" event={"ID":"672a1ec4-2d4f-4a6f-8e4a-369b6be336c7","Type":"ContainerStarted","Data":"8f75bbc23fe18d5eec02fa14bccd8964535c6ebeb72ee3ac079c0a99451077e9"} Oct 30 16:00:55.133553 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.133527 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" event={"ID":"6cd43595-43d2-4f68-85bb-fea131b69e72","Type":"ContainerStarted","Data":"e7b9a6a5a38731e9d668d2de8aa816d56e13f8faad3d35d5e45bdba731e91c83"} Oct 30 16:00:55.134143 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.134068 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" event={"ID":"6cd43595-43d2-4f68-85bb-fea131b69e72","Type":"ContainerStarted","Data":"a0dc227706094548dcf33af96f4fd91948e627a8ec3be0f5af244594b3d5d431"} Oct 30 16:00:55.135640 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.135617 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" event={"ID":"636add60-0495-48b7-bd72-1cebe229fa0c","Type":"ContainerStarted","Data":"0825e41c980bd59046f49d3ca0fcc9557615d7bff9f4641976e04b77d6b429d0"} Oct 30 16:00:55.135771 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.135758 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" event={"ID":"636add60-0495-48b7-bd72-1cebe229fa0c","Type":"ContainerStarted","Data":"fe69a768af9ac7167ce6a57ec916de90e90ec48ad2b33e4a692426f5e6db2585"} Oct 30 16:00:55.139020 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.138893 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5964675dcd-kxw7t" event={"ID":"222d4524-f967-4258-bb9c-9252db660785","Type":"ContainerStarted","Data":"b42287a1fbe24471ad38ba2930e3bea63898fdc5bcb741c0ee512e0074413b3e"} Oct 30 16:00:55.139020 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.138948 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:00:55.140735 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.140714 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" event={"ID":"8c6639c2-1eac-4716-8a14-cd33938b1c67","Type":"ContainerStarted","Data":"360bf218b27c1cb31b31fb23163bd3e3eeee1e3b05dabb0e1c4364387d3e7620"} Oct 30 16:00:55.140991 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.140976 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" event={"ID":"8c6639c2-1eac-4716-8a14-cd33938b1c67","Type":"ContainerStarted","Data":"54e1a59e7ec4b35f270ef8fa31fe598a1636e8a47620f52f7e16d69e514aee9a"} Oct 30 16:00:55.185564 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.185507 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podStartSLOduration=1.185491562 podStartE2EDuration="1.185491562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:55.180899355 +0000 UTC m=+1363.880172513" watchObservedRunningTime="2026-10-30 16:00:55.185491562 +0000 UTC m=+1363.884764712" Oct 30 16:00:55.234566 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.234518 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podStartSLOduration=1.234505294 podStartE2EDuration="1.234505294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:55.234149577 +0000 UTC m=+1363.933422736" watchObservedRunningTime="2026-10-30 16:00:55.234505294 +0000 UTC m=+1363.933778451" Oct 30 16:00:55.289577 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.289505 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podStartSLOduration=1.289482088 podStartE2EDuration="1.289482088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:55.28524506 +0000 UTC m=+1363.984518229" watchObservedRunningTime="2026-10-30 16:00:55.289482088 +0000 UTC m=+1363.988755246" Oct 30 16:00:55.349583 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.349508 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5964675dcd-kxw7t" podStartSLOduration=1.942485588 podStartE2EDuration="2.349486514s" totalImagesPullingTime="407.000926ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:55.347055082 +0000 UTC m=+1364.046328239" watchObservedRunningTime="2026-10-30 16:00:55.349486514 +0000 UTC m=+1364.048759669" Oct 30 16:00:55.508939 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.508206 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:00:55.521883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.514967 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:00:55.521883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.515043 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:00:55.544357 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.544312 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:00:55.545627 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.545562 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:00:55.545850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.545795 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:00:55.546073 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.546033 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:00:55.550096 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.549979 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:00:55.550096 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.550052 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:00:55.561864 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:55.558935 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:00:56.040881 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.034166 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podStartSLOduration=2.03413832 podStartE2EDuration="2.03413832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:00:55.423526541 +0000 UTC m=+1364.122799700" watchObservedRunningTime="2026-10-30 16:00:56.03413832 +0000 UTC m=+1364.733411478" Oct 30 16:00:56.118850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.117871 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp"] Oct 30 16:00:56.118850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.117962 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77233428-cz9mp"] Oct 30 16:00:56.508846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.507783 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:00:56.508846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.507880 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:00:56.536848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.534604 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:00:56.536848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.534669 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:00:56.546854 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.546207 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:00:56.546854 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.546274 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:00:56.558856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.557420 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:00:56.558856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:56.557483 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 16:00:57.457891 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.451551 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af60c524-888d-4f3c-8ea0-40899b674491" path="/var/lib/kubelet/pods/af60c524-888d-4f3c-8ea0-40899b674491/volumes" Oct 30 16:00:57.509420 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.509380 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:00:57.510133 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.510083 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:00:57.536162 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.536127 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:00:57.536448 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.536404 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:00:57.547409 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.547368 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:00:57.547728 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.547683 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:00:57.558059 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.557926 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:00:57.558059 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:57.557986 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 16:00:58.062976 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.062890 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b"] Oct 30 16:00:58.092980 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.092890 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfe6cc45-6hf9b"] Oct 30 16:00:58.507851 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.507344 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:00:58.507851 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.507414 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:00:58.537850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.537371 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:00:58.537850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.537453 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:00:58.563856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.548396 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:00:58.563856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.548476 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:00:58.563856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.557070 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:00:58.563856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:58.557143 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 16:00:59.521019 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.520972 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:00:59.524949 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.524899 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:00:59.528939 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.525424 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1b189e-222c-42ee-827c-55653cc72209" path="/var/lib/kubelet/pods/9c1b189e-222c-42ee-827c-55653cc72209/volumes" Oct 30 16:00:59.543514 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.543474 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:00:59.544352 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.544312 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:00:59.544811 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.544792 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:00:59.544989 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.544965 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:00:59.557329 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.557283 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:00:59.557619 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:00:59.557589 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 16:01:00.518312 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.518268 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:01:00.542000 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.541945 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:01:00.550519 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.550383 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:01:00.550519 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.550452 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:01:00.555263 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.550002 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:01:00.555410 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.555386 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:01:00.560023 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.559984 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:01:00.560303 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:00.560278 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 16:01:01.507209 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.507173 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:01:01.507453 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.507432 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:01:01.534527 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.534486 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:01:01.534708 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.534540 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:01:01.544187 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.544153 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:01:01.544308 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.544217 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:01:01.557322 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.557277 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:01:01.557951 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:01.557915 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 16:01:02.507334 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.507285 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 16:01:02.507537 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.507365 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 16:01:02.534645 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.534600 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 16:01:02.534889 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.534666 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 16:01:02.544268 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.544228 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 16:01:02.544484 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.544295 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 16:01:02.556458 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.556412 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 30 16:01:02.556672 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:02.556484 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 30 16:01:04.416860 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.416492 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:01:04.418092 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.417971 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:01:04.419792 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.419762 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" Oct 30 16:01:04.459515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.459468 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:01:04.460248 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.460222 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:01:04.461019 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.460996 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" Oct 30 16:01:04.552861 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.537843 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:01:04.552861 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.537907 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:01:04.578644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.578609 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:01:04.579446 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.579418 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:01:04.580397 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.580369 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" Oct 30 16:01:04.608983 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.608940 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:01:04.609209 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.609005 4401 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:01:04.684173 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.684086 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:01:04.685274 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.684864 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:01:04.686592 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:04.686560 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" Oct 30 16:01:06.226224 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:06.226188 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5964675dcd-kxw7t" Oct 30 16:01:09.040428 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:09.040383 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl"] Oct 30 16:01:09.053634 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:09.053597 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-159a0286-nncgl"] Oct 30 16:01:09.428881 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:09.428842 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3913ff2e-83fd-4a5f-ada0-81d2eb01efaf" path="/var/lib/kubelet/pods/3913ff2e-83fd-4a5f-ada0-81d2eb01efaf/volumes" Oct 30 16:01:12.374433 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:12.374396 4401 scope.go:122] "RemoveContainer" containerID="5f0e24f25bb4f3e785232045d1736da47bfbb630dde4f8d936510edd48ad1e47" Oct 30 16:01:12.384946 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:12.384919 4401 scope.go:122] "RemoveContainer" containerID="37783ce8b5fea084c85201084ac0edc1aa1b61bd58a7160195a36ab764d98b25" Oct 30 16:01:12.394940 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:12.394912 4401 scope.go:122] "RemoveContainer" containerID="5699e061caff465211ed2d3269b3ba82f3ddf9bea838e750e38e11c73fad2723" Oct 30 16:01:12.404694 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:12.404670 4401 scope.go:122] "RemoveContainer" containerID="fd4110e4ebe6263fdeff7f3d19c7e96c87ccd0c2108cfc5ee72a9314cff4fb93" Oct 30 16:01:14.034115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:14.034076 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm"] Oct 30 16:01:14.046872 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:14.046840 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34844f41-zl4vm"] Oct 30 16:01:15.427793 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:15.427758 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7772b8a2-b1a1-484e-ba72-7d5ec5662def" path="/var/lib/kubelet/pods/7772b8a2-b1a1-484e-ba72-7d5ec5662def/volumes" Oct 30 16:01:21.965371 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:01:21.965332 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:01:24.685895 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:24.685784 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:01:27.058987 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:27.058950 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7"] Oct 30 16:01:27.100524 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:27.069891 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn"] Oct 30 16:01:27.100776 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:27.100759 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07a0c838-lhsw7"] Oct 30 16:01:27.116115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:27.116087 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cff3d958-ppskn"] Oct 30 16:01:27.450851 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:27.445211 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ed217d-339e-45bb-ac01-3db35e7b8b90" path="/var/lib/kubelet/pods/b6ed217d-339e-45bb-ac01-3db35e7b8b90/volumes" Oct 30 16:01:27.456435 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:27.456400 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0" path="/var/lib/kubelet/pods/c9ace5cb-3ee4-4368-933f-3cd5ea1f92a0/volumes" Oct 30 16:01:29.033999 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:29.033963 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658"] Oct 30 16:01:29.047658 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:29.047628 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cf22e7a-pm658"] Oct 30 16:01:29.426970 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:29.426933 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8320b0ca-66cc-45db-b79c-2e18812e474c" path="/var/lib/kubelet/pods/8320b0ca-66cc-45db-b79c-2e18812e474c/volumes" Oct 30 16:01:30.612358 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.612324 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm"] Oct 30 16:01:30.617776 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.617751 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.633512 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.633484 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm"] Oct 30 16:01:30.642855 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.642818 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:01:30.642855 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.642817 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:01:30.643027 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.642868 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:01:30.648486 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.648463 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:01:30.774554 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.774522 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.774725 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.774585 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkkws\" (UniqueName: \"kubernetes.io/projected/0106a30e-8004-451b-8c4c-70f31b5da50b-kube-api-access-wkkws\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.774725 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.774692 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.875693 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.875610 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.875868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.875701 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkkws\" (UniqueName: \"kubernetes.io/projected/0106a30e-8004-451b-8c4c-70f31b5da50b-kube-api-access-wkkws\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.875868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.875750 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.876320 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.876291 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.878129 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.878108 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.901275 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.901249 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkkws\" (UniqueName: \"kubernetes.io/projected/0106a30e-8004-451b-8c4c-70f31b5da50b-kube-api-access-wkkws\") pod \"maas-api-revoke-sub-c1e271a9-7fvzm\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:30.932101 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:30.932084 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:31.244538 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:01:31.244497 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0106a30e_8004_451b_8c4c_70f31b5da50b.slice/crio-3ea6dc4a5ea345cd9965e48896abb852feb8a2ac4082375c3555da4684044fe4 WatchSource:0}: Error finding container 3ea6dc4a5ea345cd9965e48896abb852feb8a2ac4082375c3555da4684044fe4: Status 404 returned error can't find the container with id 3ea6dc4a5ea345cd9965e48896abb852feb8a2ac4082375c3555da4684044fe4 Oct 30 16:01:31.246066 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:31.245980 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm"] Oct 30 16:01:31.865656 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:31.865615 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" event={"ID":"0106a30e-8004-451b-8c4c-70f31b5da50b","Type":"ContainerStarted","Data":"f05ecf3d651caef7b07386b71d0829f241aad062735787f1ee622c2948f2ebd7"} Oct 30 16:01:31.866062 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:31.865674 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" event={"ID":"0106a30e-8004-451b-8c4c-70f31b5da50b","Type":"ContainerStarted","Data":"3ea6dc4a5ea345cd9965e48896abb852feb8a2ac4082375c3555da4684044fe4"} Oct 30 16:01:31.916720 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:31.916670 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" podStartSLOduration=1.916657378 podStartE2EDuration="1.916657378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:01:31.915889358 +0000 UTC m=+1400.615162516" watchObservedRunningTime="2026-10-30 16:01:31.916657378 +0000 UTC m=+1400.615930535" Oct 30 16:01:32.874933 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:32.874895 4401 generic.go:356] "Generic (PLEG): container finished" podID="0106a30e-8004-451b-8c4c-70f31b5da50b" containerID="f05ecf3d651caef7b07386b71d0829f241aad062735787f1ee622c2948f2ebd7" exitCode=0 Oct 30 16:01:32.875321 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:32.874976 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" event={"ID":"0106a30e-8004-451b-8c4c-70f31b5da50b","Type":"ContainerDied","Data":"f05ecf3d651caef7b07386b71d0829f241aad062735787f1ee622c2948f2ebd7"} Oct 30 16:01:34.024703 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.024673 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:34.107740 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.107707 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-xks-ca\") pod \"0106a30e-8004-451b-8c4c-70f31b5da50b\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " Oct 30 16:01:34.107930 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.107766 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-openshift-ca\") pod \"0106a30e-8004-451b-8c4c-70f31b5da50b\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " Oct 30 16:01:34.107930 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.107797 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0106a30e-8004-451b-8c4c-70f31b5da50b-kube-api-access-wkkws\" (UniqueName: \"kubernetes.io/projected/0106a30e-8004-451b-8c4c-70f31b5da50b-kube-api-access-wkkws\") pod \"0106a30e-8004-451b-8c4c-70f31b5da50b\" (UID: \"0106a30e-8004-451b-8c4c-70f31b5da50b\") " Oct 30 16:01:34.108152 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.108125 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-openshift-ca" pod "0106a30e-8004-451b-8c4c-70f31b5da50b" (UID: "0106a30e-8004-451b-8c4c-70f31b5da50b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:01:34.109844 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.109789 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-xks-ca" pod "0106a30e-8004-451b-8c4c-70f31b5da50b" (UID: "0106a30e-8004-451b-8c4c-70f31b5da50b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:01:34.109950 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.109869 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0106a30e-8004-451b-8c4c-70f31b5da50b-kube-api-access-wkkws" pod "0106a30e-8004-451b-8c4c-70f31b5da50b" (UID: "0106a30e-8004-451b-8c4c-70f31b5da50b"). InnerVolumeSpecName "kube-api-access-wkkws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:01:34.209022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.208947 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:01:34.209022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.208977 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkkws\" (UniqueName: \"kubernetes.io/projected/0106a30e-8004-451b-8c4c-70f31b5da50b-kube-api-access-wkkws\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:01:34.209022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.208988 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0106a30e-8004-451b-8c4c-70f31b5da50b-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:01:34.891743 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.891703 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" event={"ID":"0106a30e-8004-451b-8c4c-70f31b5da50b","Type":"ContainerDied","Data":"3ea6dc4a5ea345cd9965e48896abb852feb8a2ac4082375c3555da4684044fe4"} Oct 30 16:01:34.891941 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.891769 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea6dc4a5ea345cd9965e48896abb852feb8a2ac4082375c3555da4684044fe4" Oct 30 16:01:34.891941 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:34.891723 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm" Oct 30 16:01:35.030498 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:35.030469 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr"] Oct 30 16:01:35.033971 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:35.033949 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43d43400-9scfr"] Oct 30 16:01:35.427070 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:35.427039 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2" path="/var/lib/kubelet/pods/3cfb530f-dfd4-45ab-b7cd-03ea3459c3c2/volumes" Oct 30 16:01:41.985460 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:41.985424 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:01:42.041154 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:42.041126 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k"] Oct 30 16:01:42.055359 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:42.055331 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb"] Oct 30 16:01:42.058816 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:42.058796 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c40e426b-ftz4k"] Oct 30 16:01:42.062086 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:42.062060 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-448e74a3-q8lnb"] Oct 30 16:01:43.438333 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:43.438293 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26631d06-17b9-4ce8-80d5-ab4dcc21cf19" path="/var/lib/kubelet/pods/26631d06-17b9-4ce8-80d5-ab4dcc21cf19/volumes" Oct 30 16:01:43.439584 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:43.439551 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92f9022-bc45-4137-80ee-b0f1b36abae7" path="/var/lib/kubelet/pods/a92f9022-bc45-4137-80ee-b0f1b36abae7/volumes" Oct 30 16:01:44.040402 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:44.040364 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg"] Oct 30 16:01:44.055623 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:44.055578 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a4324a5-m4mpg"] Oct 30 16:01:45.427449 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:45.427416 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a04bd0-7ab3-450f-9139-3ab5d8993aa7" path="/var/lib/kubelet/pods/f4a04bd0-7ab3-450f-9139-3ab5d8993aa7/volumes" Oct 30 16:01:46.701632 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:46.701599 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:01:49.055815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:49.055761 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf"] Oct 30 16:01:49.102873 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:49.102841 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efc30c9e-vjxxf"] Oct 30 16:01:49.436766 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:49.436729 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07ba13d-aa01-4829-a390-2f3378833212" path="/var/lib/kubelet/pods/d07ba13d-aa01-4829-a390-2f3378833212/volumes" Oct 30 16:01:50.754151 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.754116 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5"] Oct 30 16:01:50.754603 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.754586 4401 cpu_manager.go:414] "containerMap: removing container" podUID="0106a30e-8004-451b-8c4c-70f31b5da50b" containerName="revoke-keys" Oct 30 16:01:50.754603 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.754598 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0106a30e-8004-451b-8c4c-70f31b5da50b" containerName="revoke-keys" Oct 30 16:01:50.754707 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.754684 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="0106a30e-8004-451b-8c4c-70f31b5da50b" containerName="revoke-keys" Oct 30 16:01:50.759249 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.759231 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.775879 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.775852 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5"] Oct 30 16:01:50.785874 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.785844 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:01:50.786004 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.785987 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:01:50.786052 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.786034 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:01:50.789862 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.789845 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:01:50.849362 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.849325 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf5n4\" (UniqueName: \"kubernetes.io/projected/050e229e-af32-4e27-aba9-b143cb16c679-kube-api-access-xf5n4\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.849512 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.849459 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.849553 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.849512 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.950784 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.950743 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.950990 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.950797 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf5n4\" (UniqueName: \"kubernetes.io/projected/050e229e-af32-4e27-aba9-b143cb16c679-kube-api-access-xf5n4\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.950990 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.950901 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.951496 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.951470 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.953064 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.953044 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:50.976008 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:50.975979 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf5n4\" (UniqueName: \"kubernetes.io/projected/050e229e-af32-4e27-aba9-b143cb16c679-kube-api-access-xf5n4\") pod \"maas-api-revoke-sub-2e1f5b5f-vw9s5\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:51.074021 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:51.073943 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:51.384146 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:51.384113 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5"] Oct 30 16:01:51.384331 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:01:51.384310 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050e229e_af32_4e27_aba9_b143cb16c679.slice/crio-a44472d9a8786adae7f9f1ba82732a6d3268c37025dcc6dbacd884c65c479a57 WatchSource:0}: Error finding container a44472d9a8786adae7f9f1ba82732a6d3268c37025dcc6dbacd884c65c479a57: Status 404 returned error can't find the container with id a44472d9a8786adae7f9f1ba82732a6d3268c37025dcc6dbacd884c65c479a57 Oct 30 16:01:51.966416 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:01:51.966338 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:01:52.118910 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:52.118875 4401 generic.go:356] "Generic (PLEG): container finished" podID="050e229e-af32-4e27-aba9-b143cb16c679" containerID="34ff8e3c9c38041c29aaa5029085afc34d55cdc067c826376c2dc156e25934a8" exitCode=0 Oct 30 16:01:52.119034 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:52.118925 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" event={"ID":"050e229e-af32-4e27-aba9-b143cb16c679","Type":"ContainerDied","Data":"34ff8e3c9c38041c29aaa5029085afc34d55cdc067c826376c2dc156e25934a8"} Oct 30 16:01:52.119034 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:52.119005 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" event={"ID":"050e229e-af32-4e27-aba9-b143cb16c679","Type":"ContainerStarted","Data":"a44472d9a8786adae7f9f1ba82732a6d3268c37025dcc6dbacd884c65c479a57"} Oct 30 16:01:53.258983 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.258961 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:53.271646 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.271620 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-xks-ca\") pod \"050e229e-af32-4e27-aba9-b143cb16c679\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " Oct 30 16:01:53.271778 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.271747 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-openshift-ca\") pod \"050e229e-af32-4e27-aba9-b143cb16c679\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " Oct 30 16:01:53.271881 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.271864 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/050e229e-af32-4e27-aba9-b143cb16c679-kube-api-access-xf5n4\" (UniqueName: \"kubernetes.io/projected/050e229e-af32-4e27-aba9-b143cb16c679-kube-api-access-xf5n4\") pod \"050e229e-af32-4e27-aba9-b143cb16c679\" (UID: \"050e229e-af32-4e27-aba9-b143cb16c679\") " Oct 30 16:01:53.272084 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.272061 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-openshift-ca" pod "050e229e-af32-4e27-aba9-b143cb16c679" (UID: "050e229e-af32-4e27-aba9-b143cb16c679"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:01:53.272207 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.272193 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:01:53.273515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.273487 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-xks-ca" pod "050e229e-af32-4e27-aba9-b143cb16c679" (UID: "050e229e-af32-4e27-aba9-b143cb16c679"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:01:53.273701 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.273677 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050e229e-af32-4e27-aba9-b143cb16c679-kube-api-access-xf5n4" pod "050e229e-af32-4e27-aba9-b143cb16c679" (UID: "050e229e-af32-4e27-aba9-b143cb16c679"). InnerVolumeSpecName "kube-api-access-xf5n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:01:53.373615 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.373584 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf5n4\" (UniqueName: \"kubernetes.io/projected/050e229e-af32-4e27-aba9-b143cb16c679-kube-api-access-xf5n4\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:01:53.373615 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:53.373611 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/050e229e-af32-4e27-aba9-b143cb16c679-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:01:54.033006 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:54.032974 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92"] Oct 30 16:01:54.047122 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:54.047090 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-040e4fa2-56t92"] Oct 30 16:01:54.133723 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:54.133696 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" Oct 30 16:01:54.133923 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:54.133743 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5" event={"ID":"050e229e-af32-4e27-aba9-b143cb16c679","Type":"ContainerDied","Data":"a44472d9a8786adae7f9f1ba82732a6d3268c37025dcc6dbacd884c65c479a57"} Oct 30 16:01:54.133923 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:54.133794 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44472d9a8786adae7f9f1ba82732a6d3268c37025dcc6dbacd884c65c479a57" Oct 30 16:01:55.427246 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:55.427211 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afb2fdc-253a-4459-bac4-07d0a00faba4" path="/var/lib/kubelet/pods/3afb2fdc-253a-4459-bac4-07d0a00faba4/volumes" Oct 30 16:01:58.031697 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:58.031665 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc"] Oct 30 16:01:58.045207 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:58.045178 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68d04e69-ts5lc"] Oct 30 16:01:59.426867 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:01:59.426804 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e143195-565b-4ce3-8638-c7ba6b23ab27" path="/var/lib/kubelet/pods/5e143195-565b-4ce3-8638-c7ba6b23ab27/volumes" Oct 30 16:02:02.050194 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.050156 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25"] Oct 30 16:02:02.050633 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.050618 4401 cpu_manager.go:414] "containerMap: removing container" podUID="050e229e-af32-4e27-aba9-b143cb16c679" containerName="revoke-keys" Oct 30 16:02:02.050681 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.050635 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="050e229e-af32-4e27-aba9-b143cb16c679" containerName="revoke-keys" Oct 30 16:02:02.050753 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.050743 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="050e229e-af32-4e27-aba9-b143cb16c679" containerName="revoke-keys" Oct 30 16:02:02.053944 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.053928 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.070754 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.070726 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:02:02.070920 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.070799 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:02:02.074549 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.074523 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25"] Oct 30 16:02:02.084249 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.084229 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:02:02.084344 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.084293 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:02:02.160630 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.160592 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.160789 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.160704 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.160860 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.160792 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5cs\" (UniqueName: \"kubernetes.io/projected/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-kube-api-access-pd5cs\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.261628 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.261602 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5cs\" (UniqueName: \"kubernetes.io/projected/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-kube-api-access-pd5cs\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.261789 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.261653 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.261789 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.261737 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.262446 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.262424 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.264030 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.264011 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.287922 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.287899 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5cs\" (UniqueName: \"kubernetes.io/projected/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-kube-api-access-pd5cs\") pod \"maas-api-revoke-sub-ec843018-bfw25\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.366946 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.366923 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:02.671000 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:02.670973 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25"] Oct 30 16:02:02.672021 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:02:02.671992 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8423968c_a0fb_45b6_9c08_6d72d6c8bcf3.slice/crio-cb0f5f62fb8b49d499f497b6256b1c95bcbf390f5adbaad3926115304eeca785 WatchSource:0}: Error finding container cb0f5f62fb8b49d499f497b6256b1c95bcbf390f5adbaad3926115304eeca785: Status 404 returned error can't find the container with id cb0f5f62fb8b49d499f497b6256b1c95bcbf390f5adbaad3926115304eeca785 Oct 30 16:02:03.009632 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:03.009594 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:02:03.198984 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:03.198949 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" event={"ID":"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3","Type":"ContainerStarted","Data":"162b751aaa835561f864cfb40be6409d4d46bf872b7f14acf1e165a76a22f216"} Oct 30 16:02:03.199352 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:03.199033 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" event={"ID":"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3","Type":"ContainerStarted","Data":"cb0f5f62fb8b49d499f497b6256b1c95bcbf390f5adbaad3926115304eeca785"} Oct 30 16:02:03.245390 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:03.245303 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" podStartSLOduration=1.245290095 podStartE2EDuration="1.245290095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:02:03.243065066 +0000 UTC m=+1431.942338216" watchObservedRunningTime="2026-10-30 16:02:03.245290095 +0000 UTC m=+1431.944563253" Oct 30 16:02:04.213377 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:04.213332 4401 generic.go:356] "Generic (PLEG): container finished" podID="8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" containerID="162b751aaa835561f864cfb40be6409d4d46bf872b7f14acf1e165a76a22f216" exitCode=0 Oct 30 16:02:04.213937 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:04.213443 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" event={"ID":"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3","Type":"ContainerDied","Data":"162b751aaa835561f864cfb40be6409d4d46bf872b7f14acf1e165a76a22f216"} Oct 30 16:02:05.516043 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.515924 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:05.516658 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.516624 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:05.517271 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.517249 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:05.517444 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.517423 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:05.737109 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.737071 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:05.737397 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.737364 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:05.746846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.739539 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:05.746846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:05.739600 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:06.088803 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.088769 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:06.155646 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.155608 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-openshift-ca\") pod \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " Oct 30 16:02:06.161139 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.161114 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-xks-ca\") pod \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " Oct 30 16:02:06.161709 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.161689 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-kube-api-access-pd5cs\" (UniqueName: \"kubernetes.io/projected/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-kube-api-access-pd5cs\") pod \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\" (UID: \"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3\") " Oct 30 16:02:06.162543 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.161019 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-openshift-ca" pod "8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" (UID: "8423968c-a0fb-45b6-9c08-6d72d6c8bcf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:02:06.184903 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.177426 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-kube-api-access-pd5cs" pod "8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" (UID: "8423968c-a0fb-45b6-9c08-6d72d6c8bcf3"). InnerVolumeSpecName "kube-api-access-pd5cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:02:06.232846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.222943 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-xks-ca" pod "8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" (UID: "8423968c-a0fb-45b6-9c08-6d72d6c8bcf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:02:06.274808 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.274538 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:06.274808 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.274574 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:06.274808 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.274590 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd5cs\" (UniqueName: \"kubernetes.io/projected/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3-kube-api-access-pd5cs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:06.343975 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.343940 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" event={"ID":"8423968c-a0fb-45b6-9c08-6d72d6c8bcf3","Type":"ContainerDied","Data":"cb0f5f62fb8b49d499f497b6256b1c95bcbf390f5adbaad3926115304eeca785"} Oct 30 16:02:06.344229 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.344208 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb0f5f62fb8b49d499f497b6256b1c95bcbf390f5adbaad3926115304eeca785" Oct 30 16:02:06.344416 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:06.344400 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25" Oct 30 16:02:09.050451 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:09.050411 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs"] Oct 30 16:02:09.066389 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:09.066357 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05a14e06-l8gbs"] Oct 30 16:02:09.072514 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:09.072494 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch"] Oct 30 16:02:09.078038 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:09.078009 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13c91299-m79ch"] Oct 30 16:02:09.175644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:09.175617 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:02:09.426641 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:09.426598 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc258d9-c4f1-40c5-ab67-0fe3d613698f" path="/var/lib/kubelet/pods/cfc258d9-c4f1-40c5-ab67-0fe3d613698f/volumes" Oct 30 16:02:09.427117 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:09.427094 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5e23559-203e-42f8-84a6-8e9bb10ad432" path="/var/lib/kubelet/pods/f5e23559-203e-42f8-84a6-8e9bb10ad432/volumes" Oct 30 16:02:12.487956 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.487924 4401 scope.go:122] "RemoveContainer" containerID="05f98370ea814e3227e916f8470c32a7f7ae813af254860ac11089cf2e374da2" Oct 30 16:02:12.497918 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.497896 4401 scope.go:122] "RemoveContainer" containerID="f5f894d0af9bf3f121043180408e3cc0e5764084e0af7930694ff105e3dfb736" Oct 30 16:02:12.506628 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.506598 4401 scope.go:122] "RemoveContainer" containerID="54401bea0a911e95f8683b661f740bb74cd4371eb30738197b51be72ad628246" Oct 30 16:02:12.515054 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.515032 4401 scope.go:122] "RemoveContainer" containerID="e4246b2a22dbf7898bab8fcb652a0d2d769f4338349515ebbecd4dcfb1d112fd" Oct 30 16:02:12.523182 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.523164 4401 scope.go:122] "RemoveContainer" containerID="473f5bb960c77d85d040140f35dcca0287daaf0d78e4de5ab8088b74b68bd84b" Oct 30 16:02:12.531909 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.531884 4401 scope.go:122] "RemoveContainer" containerID="7c04c702091d22729d7b27bd068437012922d66798e86e1772af3649ba5df546" Oct 30 16:02:12.539722 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.539704 4401 scope.go:122] "RemoveContainer" containerID="3d9e4ae7d8a5e8f27bbaaf4e5f321d50bf7fbba3b728cf5907f011a31096a636" Oct 30 16:02:12.547800 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.547777 4401 scope.go:122] "RemoveContainer" containerID="ced473035fb61d2e525e1a4e5aad2db37de180d2b8362f3ca32fe778977bcd8e" Oct 30 16:02:12.556718 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:12.556703 4401 scope.go:122] "RemoveContainer" containerID="eb65710cb67441b094c60311d04b21fef89e7edc34543d47256903f012a11231" Oct 30 16:02:17.048150 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:17.048115 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl"] Oct 30 16:02:17.061546 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:17.061517 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a59c045e-hzkzl"] Oct 30 16:02:17.426905 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:17.426872 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aafde56-f3e7-415b-83b7-98c52e390add" path="/var/lib/kubelet/pods/9aafde56-f3e7-415b-83b7-98c52e390add/volumes" Oct 30 16:02:17.918514 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:17.918473 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:02:20.180922 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.180876 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm"] Oct 30 16:02:20.235129 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.235082 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd6546f-qthjm"] Oct 30 16:02:20.473019 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.472926 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:20.473323 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.473289 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:20.520849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.519925 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:20.520849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.520034 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:20.637849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.633916 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:20.637849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.634017 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:20.734021 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.733907 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:02:20.734312 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:20.734261 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:02:21.428373 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:21.428341 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faca6252-0610-4f50-aed9-bd07ef8c7938" path="/var/lib/kubelet/pods/faca6252-0610-4f50-aed9-bd07ef8c7938/volumes" Oct 30 16:02:21.966136 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:02:21.966101 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:02:22.031407 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:22.031367 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq"] Oct 30 16:02:22.034530 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:22.034504 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa2ac439-gxfwq"] Oct 30 16:02:23.427012 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:23.426981 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a40a3f-28dd-4ff7-ae48-33c72d796a21" path="/var/lib/kubelet/pods/06a40a3f-28dd-4ff7-ae48-33c72d796a21/volumes" Oct 30 16:02:30.067598 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.067560 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5"] Oct 30 16:02:30.068085 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.068029 4401 cpu_manager.go:414] "containerMap: removing container" podUID="8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" containerName="revoke-keys" Oct 30 16:02:30.068085 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.068042 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" containerName="revoke-keys" Oct 30 16:02:30.068167 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.068140 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" containerName="revoke-keys" Oct 30 16:02:30.073538 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.073518 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.088814 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.088785 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5"] Oct 30 16:02:30.098928 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.098900 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:02:30.098928 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.098901 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:02:30.099079 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.098934 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:02:30.105003 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.104981 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:02:30.195426 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.195397 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk9bw\" (UniqueName: \"kubernetes.io/projected/20689837-6d7a-4b07-9d21-039f11a0f146-kube-api-access-dk9bw\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.195574 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.195437 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.195649 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.195588 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.296184 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.296157 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.296309 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.296231 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk9bw\" (UniqueName: \"kubernetes.io/projected/20689837-6d7a-4b07-9d21-039f11a0f146-kube-api-access-dk9bw\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.296309 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.296253 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.296871 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.296851 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.298408 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.298387 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.321176 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.321129 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk9bw\" (UniqueName: \"kubernetes.io/projected/20689837-6d7a-4b07-9d21-039f11a0f146-kube-api-access-dk9bw\") pod \"maas-api-revoke-sub-046e1343-w4md5\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.387387 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.387361 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:30.696021 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:30.695999 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5"] Oct 30 16:02:30.696741 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:02:30.696709 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20689837_6d7a_4b07_9d21_039f11a0f146.slice/crio-bd03dd1d77d6abfd8e7af742d996722bd115f0a6fa8faa889964963a4b0d9825 WatchSource:0}: Error finding container bd03dd1d77d6abfd8e7af742d996722bd115f0a6fa8faa889964963a4b0d9825: Status 404 returned error can't find the container with id bd03dd1d77d6abfd8e7af742d996722bd115f0a6fa8faa889964963a4b0d9825 Oct 30 16:02:31.570882 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:31.570851 4401 generic.go:356] "Generic (PLEG): container finished" podID="20689837-6d7a-4b07-9d21-039f11a0f146" containerID="89c20e57532f30636928cc86320d770d44b90c8593983171ace3aa012c41de3b" exitCode=0 Oct 30 16:02:31.571237 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:31.570943 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" event={"ID":"20689837-6d7a-4b07-9d21-039f11a0f146","Type":"ContainerDied","Data":"89c20e57532f30636928cc86320d770d44b90c8593983171ace3aa012c41de3b"} Oct 30 16:02:31.571237 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:31.570999 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" event={"ID":"20689837-6d7a-4b07-9d21-039f11a0f146","Type":"ContainerStarted","Data":"bd03dd1d77d6abfd8e7af742d996722bd115f0a6fa8faa889964963a4b0d9825"} Oct 30 16:02:32.717971 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.717947 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:32.822847 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.822795 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-xks-ca\") pod \"20689837-6d7a-4b07-9d21-039f11a0f146\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " Oct 30 16:02:32.823016 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.822923 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-openshift-ca\") pod \"20689837-6d7a-4b07-9d21-039f11a0f146\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " Oct 30 16:02:32.823060 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.823031 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20689837-6d7a-4b07-9d21-039f11a0f146-kube-api-access-dk9bw\" (UniqueName: \"kubernetes.io/projected/20689837-6d7a-4b07-9d21-039f11a0f146-kube-api-access-dk9bw\") pod \"20689837-6d7a-4b07-9d21-039f11a0f146\" (UID: \"20689837-6d7a-4b07-9d21-039f11a0f146\") " Oct 30 16:02:32.823336 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.823307 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-openshift-ca" pod "20689837-6d7a-4b07-9d21-039f11a0f146" (UID: "20689837-6d7a-4b07-9d21-039f11a0f146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:02:32.823475 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.823462 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:32.824788 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.824770 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-xks-ca" pod "20689837-6d7a-4b07-9d21-039f11a0f146" (UID: "20689837-6d7a-4b07-9d21-039f11a0f146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:02:32.824945 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.824924 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20689837-6d7a-4b07-9d21-039f11a0f146-kube-api-access-dk9bw" pod "20689837-6d7a-4b07-9d21-039f11a0f146" (UID: "20689837-6d7a-4b07-9d21-039f11a0f146"). InnerVolumeSpecName "kube-api-access-dk9bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:02:32.924670 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.924626 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20689837-6d7a-4b07-9d21-039f11a0f146-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:32.924670 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:32.924664 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk9bw\" (UniqueName: \"kubernetes.io/projected/20689837-6d7a-4b07-9d21-039f11a0f146-kube-api-access-dk9bw\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:33.585962 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:33.585923 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" event={"ID":"20689837-6d7a-4b07-9d21-039f11a0f146","Type":"ContainerDied","Data":"bd03dd1d77d6abfd8e7af742d996722bd115f0a6fa8faa889964963a4b0d9825"} Oct 30 16:02:33.585962 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:33.585945 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5" Oct 30 16:02:33.586163 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:33.585974 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd03dd1d77d6abfd8e7af742d996722bd115f0a6fa8faa889964963a4b0d9825" Oct 30 16:02:34.041208 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:34.041171 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb"] Oct 30 16:02:34.058082 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:34.058046 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6a50dd6-nmwvb"] Oct 30 16:02:35.426815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:35.426780 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b35df3-109f-446b-8115-c12d691afd6d" path="/var/lib/kubelet/pods/89b35df3-109f-446b-8115-c12d691afd6d/volumes" Oct 30 16:02:40.421349 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.421312 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd"] Oct 30 16:02:40.421810 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.421794 4401 cpu_manager.go:414] "containerMap: removing container" podUID="20689837-6d7a-4b07-9d21-039f11a0f146" containerName="revoke-keys" Oct 30 16:02:40.421882 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.421811 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20689837-6d7a-4b07-9d21-039f11a0f146" containerName="revoke-keys" Oct 30 16:02:40.422030 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.421922 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="20689837-6d7a-4b07-9d21-039f11a0f146" containerName="revoke-keys" Oct 30 16:02:40.426203 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.426187 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.447393 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.447368 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:02:40.447520 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.447404 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:02:40.447995 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.447974 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:02:40.461879 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.461858 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:02:40.467642 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.467619 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd"] Oct 30 16:02:40.498285 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.498253 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwghm\" (UniqueName: \"kubernetes.io/projected/e9322250-35e6-428f-bcdf-67a08f4af1d3-kube-api-access-jwghm\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.498427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.498407 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.498514 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.498435 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.598963 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.598936 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwghm\" (UniqueName: \"kubernetes.io/projected/e9322250-35e6-428f-bcdf-67a08f4af1d3-kube-api-access-jwghm\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.599078 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.598986 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.599078 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.599010 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.599574 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.599556 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.601268 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.601248 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.625265 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.625244 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwghm\" (UniqueName: \"kubernetes.io/projected/e9322250-35e6-428f-bcdf-67a08f4af1d3-kube-api-access-jwghm\") pod \"maas-api-revoke-sub-d3fba056-vltjd\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:40.739847 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:40.739762 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:41.044390 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:41.044363 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd"] Oct 30 16:02:41.045121 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:02:41.045094 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9322250_35e6_428f_bcdf_67a08f4af1d3.slice/crio-a913a65efdb5df9eeee5e67986c441016266ec6fe102aeca2c17b4e61af3d32a WatchSource:0}: Error finding container a913a65efdb5df9eeee5e67986c441016266ec6fe102aeca2c17b4e61af3d32a: Status 404 returned error can't find the container with id a913a65efdb5df9eeee5e67986c441016266ec6fe102aeca2c17b4e61af3d32a Oct 30 16:02:41.402553 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:41.402526 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:02:41.641535 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:41.641454 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" event={"ID":"e9322250-35e6-428f-bcdf-67a08f4af1d3","Type":"ContainerStarted","Data":"fec69a83107e894780db77e646c48373aacf17bf1dbc763e3a5a8c0bd6c73b22"} Oct 30 16:02:41.641535 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:41.641516 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" event={"ID":"e9322250-35e6-428f-bcdf-67a08f4af1d3","Type":"ContainerStarted","Data":"a913a65efdb5df9eeee5e67986c441016266ec6fe102aeca2c17b4e61af3d32a"} Oct 30 16:02:41.696781 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:41.696734 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" podStartSLOduration=1.69672072 podStartE2EDuration="1.69672072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:02:41.696314304 +0000 UTC m=+1470.395587547" watchObservedRunningTime="2026-10-30 16:02:41.69672072 +0000 UTC m=+1470.395993877" Oct 30 16:02:42.664925 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:42.664872 4401 generic.go:356] "Generic (PLEG): container finished" podID="e9322250-35e6-428f-bcdf-67a08f4af1d3" containerID="fec69a83107e894780db77e646c48373aacf17bf1dbc763e3a5a8c0bd6c73b22" exitCode=0 Oct 30 16:02:42.665716 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:42.665676 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" event={"ID":"e9322250-35e6-428f-bcdf-67a08f4af1d3","Type":"ContainerDied","Data":"fec69a83107e894780db77e646c48373aacf17bf1dbc763e3a5a8c0bd6c73b22"} Oct 30 16:02:44.157274 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.157241 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s"] Oct 30 16:02:44.195913 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.195874 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr"] Oct 30 16:02:44.262727 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.262692 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d818911-ct67s"] Oct 30 16:02:44.268771 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.268730 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a0931d3-zkgmr"] Oct 30 16:02:44.327407 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.327095 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:44.433277 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.432443 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-xks-ca\") pod \"e9322250-35e6-428f-bcdf-67a08f4af1d3\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " Oct 30 16:02:44.433277 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.432503 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9322250-35e6-428f-bcdf-67a08f4af1d3-kube-api-access-jwghm\" (UniqueName: \"kubernetes.io/projected/e9322250-35e6-428f-bcdf-67a08f4af1d3-kube-api-access-jwghm\") pod \"e9322250-35e6-428f-bcdf-67a08f4af1d3\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " Oct 30 16:02:44.433277 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.432575 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-openshift-ca\") pod \"e9322250-35e6-428f-bcdf-67a08f4af1d3\" (UID: \"e9322250-35e6-428f-bcdf-67a08f4af1d3\") " Oct 30 16:02:44.433876 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.433852 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-openshift-ca" pod "e9322250-35e6-428f-bcdf-67a08f4af1d3" (UID: "e9322250-35e6-428f-bcdf-67a08f4af1d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:02:44.438177 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.438154 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-xks-ca" pod "e9322250-35e6-428f-bcdf-67a08f4af1d3" (UID: "e9322250-35e6-428f-bcdf-67a08f4af1d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:02:44.438620 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.438584 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9322250-35e6-428f-bcdf-67a08f4af1d3-kube-api-access-jwghm" pod "e9322250-35e6-428f-bcdf-67a08f4af1d3" (UID: "e9322250-35e6-428f-bcdf-67a08f4af1d3"). InnerVolumeSpecName "kube-api-access-jwghm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:02:44.533687 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.533650 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:44.533687 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.533690 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwghm\" (UniqueName: \"kubernetes.io/projected/e9322250-35e6-428f-bcdf-67a08f4af1d3-kube-api-access-jwghm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:44.533955 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.533706 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9322250-35e6-428f-bcdf-67a08f4af1d3-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:02:44.709942 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.709850 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" event={"ID":"e9322250-35e6-428f-bcdf-67a08f4af1d3","Type":"ContainerDied","Data":"a913a65efdb5df9eeee5e67986c441016266ec6fe102aeca2c17b4e61af3d32a"} Oct 30 16:02:44.709942 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.709900 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a913a65efdb5df9eeee5e67986c441016266ec6fe102aeca2c17b4e61af3d32a" Oct 30 16:02:44.709942 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:44.709864 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd" Oct 30 16:02:45.427243 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:45.427211 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26b94c73-08da-4cb8-bd5f-1e5b6eef176f" path="/var/lib/kubelet/pods/26b94c73-08da-4cb8-bd5f-1e5b6eef176f/volumes" Oct 30 16:02:45.427685 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:45.427630 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45" path="/var/lib/kubelet/pods/4d4dfe6b-39ff-4a6e-a6c9-32e24a8edd45/volumes" Oct 30 16:02:51.780916 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:51.780884 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:02:51.967018 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:02:51.966976 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:02:53.030875 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:53.030844 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n"] Oct 30 16:02:53.045208 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:53.045178 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d12b81-q2w6n"] Oct 30 16:02:53.426570 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:53.426537 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3bd806c-2fa7-43eb-92f8-0997e09a6627" path="/var/lib/kubelet/pods/e3bd806c-2fa7-43eb-92f8-0997e09a6627/volumes" Oct 30 16:02:55.038165 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:55.038089 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b"] Oct 30 16:02:55.051969 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:55.051940 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-873e56eb-w4x4b"] Oct 30 16:02:55.427213 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:55.427175 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb20672-10c1-4da9-99e2-67f3896c83ec" path="/var/lib/kubelet/pods/1eb20672-10c1-4da9-99e2-67f3896c83ec/volumes" Oct 30 16:02:56.527633 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:02:56.527595 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:03:02.176587 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.176544 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst"] Oct 30 16:03:02.177073 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.177035 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e9322250-35e6-428f-bcdf-67a08f4af1d3" containerName="revoke-keys" Oct 30 16:03:02.177073 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.177048 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9322250-35e6-428f-bcdf-67a08f4af1d3" containerName="revoke-keys" Oct 30 16:03:02.177148 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.177140 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9322250-35e6-428f-bcdf-67a08f4af1d3" containerName="revoke-keys" Oct 30 16:03:02.180965 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.180945 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.200884 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.200862 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst"] Oct 30 16:03:02.210704 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.210679 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:03:02.210876 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.210717 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:03:02.210876 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.210745 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:03:02.215522 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.215497 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:03:02.227383 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.227352 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.227383 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.227380 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wkpg\" (UniqueName: \"kubernetes.io/projected/ba205c44-4823-4b48-b193-61b74f738f04-kube-api-access-6wkpg\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.227565 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.227519 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.328160 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.328133 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.328296 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.328165 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wkpg\" (UniqueName: \"kubernetes.io/projected/ba205c44-4823-4b48-b193-61b74f738f04-kube-api-access-6wkpg\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.328296 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.328261 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.328913 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.328893 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.330439 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.330421 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.353899 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.353874 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wkpg\" (UniqueName: \"kubernetes.io/projected/ba205c44-4823-4b48-b193-61b74f738f04-kube-api-access-6wkpg\") pod \"maas-api-revoke-sub-ae594979-lbjst\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.494346 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.494262 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:02.800585 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.800546 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst"] Oct 30 16:03:02.800704 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:03:02.800604 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba205c44_4823_4b48_b193_61b74f738f04.slice/crio-e2cc28dd974d80f840dc753ee18294f78886f71ed83cc0a5ecaff0ce852f97bc WatchSource:0}: Error finding container e2cc28dd974d80f840dc753ee18294f78886f71ed83cc0a5ecaff0ce852f97bc: Status 404 returned error can't find the container with id e2cc28dd974d80f840dc753ee18294f78886f71ed83cc0a5ecaff0ce852f97bc Oct 30 16:03:02.870592 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:02.870562 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" event={"ID":"ba205c44-4823-4b48-b193-61b74f738f04","Type":"ContainerStarted","Data":"e2cc28dd974d80f840dc753ee18294f78886f71ed83cc0a5ecaff0ce852f97bc"} Oct 30 16:03:03.878959 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:03.878923 4401 generic.go:356] "Generic (PLEG): container finished" podID="ba205c44-4823-4b48-b193-61b74f738f04" containerID="624ff63bda3e186dfd3e43e79f9d262cf4877b98cb314073dc1ced4bfdf7ab75" exitCode=0 Oct 30 16:03:03.879362 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:03.879015 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" event={"ID":"ba205c44-4823-4b48-b193-61b74f738f04","Type":"ContainerDied","Data":"624ff63bda3e186dfd3e43e79f9d262cf4877b98cb314073dc1ced4bfdf7ab75"} Oct 30 16:03:04.044590 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:04.044557 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54"] Oct 30 16:03:04.053316 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:04.053272 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp"] Oct 30 16:03:04.056310 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:04.056289 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9adfa703-x5s54"] Oct 30 16:03:04.058878 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:04.058857 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b744b60-6nwbp"] Oct 30 16:03:05.022535 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.022507 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:05.052644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.052619 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba205c44-4823-4b48-b193-61b74f738f04-kube-api-access-6wkpg\" (UniqueName: \"kubernetes.io/projected/ba205c44-4823-4b48-b193-61b74f738f04-kube-api-access-6wkpg\") pod \"ba205c44-4823-4b48-b193-61b74f738f04\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " Oct 30 16:03:05.052785 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.052678 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-openshift-ca\") pod \"ba205c44-4823-4b48-b193-61b74f738f04\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " Oct 30 16:03:05.052893 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.052817 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-xks-ca\") pod \"ba205c44-4823-4b48-b193-61b74f738f04\" (UID: \"ba205c44-4823-4b48-b193-61b74f738f04\") " Oct 30 16:03:05.053060 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.053034 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-openshift-ca" pod "ba205c44-4823-4b48-b193-61b74f738f04" (UID: "ba205c44-4823-4b48-b193-61b74f738f04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:03:05.053233 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.053218 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:05.054716 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.054694 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-xks-ca" pod "ba205c44-4823-4b48-b193-61b74f738f04" (UID: "ba205c44-4823-4b48-b193-61b74f738f04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:03:05.054798 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.054722 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba205c44-4823-4b48-b193-61b74f738f04-kube-api-access-6wkpg" pod "ba205c44-4823-4b48-b193-61b74f738f04" (UID: "ba205c44-4823-4b48-b193-61b74f738f04"). InnerVolumeSpecName "kube-api-access-6wkpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:03:05.154109 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.154038 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wkpg\" (UniqueName: \"kubernetes.io/projected/ba205c44-4823-4b48-b193-61b74f738f04-kube-api-access-6wkpg\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:05.154109 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.154065 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba205c44-4823-4b48-b193-61b74f738f04-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:05.426806 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.426728 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f83c136-d8fd-4d50-bf3a-9a3db8cf9403" path="/var/lib/kubelet/pods/1f83c136-d8fd-4d50-bf3a-9a3db8cf9403/volumes" Oct 30 16:03:05.427191 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.427176 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d87eb1-e2c5-4886-8345-10368cfff705" path="/var/lib/kubelet/pods/f5d87eb1-e2c5-4886-8345-10368cfff705/volumes" Oct 30 16:03:05.897087 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.897052 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" event={"ID":"ba205c44-4823-4b48-b193-61b74f738f04","Type":"ContainerDied","Data":"e2cc28dd974d80f840dc753ee18294f78886f71ed83cc0a5ecaff0ce852f97bc"} Oct 30 16:03:05.897268 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.897106 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2cc28dd974d80f840dc753ee18294f78886f71ed83cc0a5ecaff0ce852f97bc" Oct 30 16:03:05.897268 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:05.897062 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst" Oct 30 16:03:12.630274 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.630238 4401 scope.go:122] "RemoveContainer" containerID="5e99be56efb0aced5f07116f2f946b58e549da69a78ee3c6be84e0e493650452" Oct 30 16:03:12.640725 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.640702 4401 scope.go:122] "RemoveContainer" containerID="132b037633f52aba163bef69b1e2be254b6d6734950bbd2e5df48d9c6256ee43" Oct 30 16:03:12.649741 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.649715 4401 scope.go:122] "RemoveContainer" containerID="5b7b8f88365d207ff3e3a9a7eb8373972ae26a4d4164bb5849a6a6bbde2df7d2" Oct 30 16:03:12.658551 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.658531 4401 scope.go:122] "RemoveContainer" containerID="ff64d5ae884cae0a7f685b8f347529912b9c9ca5939a4e679d98e07822ae438a" Oct 30 16:03:12.667182 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.667162 4401 scope.go:122] "RemoveContainer" containerID="e17a1df9c08181682feee4e6b37b7714db8775a62c5ce1a2c7a4d0a0749cc582" Oct 30 16:03:12.675738 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.675720 4401 scope.go:122] "RemoveContainer" containerID="0d214dab73ec886783f02cdb688a34969c8dcdae126a7071f72335827e771a34" Oct 30 16:03:12.684051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.684029 4401 scope.go:122] "RemoveContainer" containerID="da11bd58f4339614243e2959b65e56d23fedb8fea1a86c23437a92596ac6e7ae" Oct 30 16:03:12.692458 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.692432 4401 scope.go:122] "RemoveContainer" containerID="18e0cab785f8096477fcd2444fe58fb93c41160d7b3d4b7f0324d9ddae1eb064" Oct 30 16:03:12.703359 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.703338 4401 scope.go:122] "RemoveContainer" containerID="058e602cf112d62b3d6883d30a0ede8b6a871b036a59a6c783cfa6e395516e1b" Oct 30 16:03:12.711701 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.711684 4401 scope.go:122] "RemoveContainer" containerID="6711f08f4c4043ef2804e28e49149d68965adc52516925c2fe611643fc7a91bd" Oct 30 16:03:12.719648 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.719630 4401 scope.go:122] "RemoveContainer" containerID="e6fae34d6ab616cc2f4cef8afae62cdd33c1b9014bfbd44e282ddce7e657edb1" Oct 30 16:03:12.727561 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:12.727540 4401 scope.go:122] "RemoveContainer" containerID="981c103ae38a263f3b2629f99407843057f62727a0256f2c0898f6b267045f7f" Oct 30 16:03:13.476364 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:13.476334 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:03:18.721080 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:18.721043 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:03:21.965718 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:03:21.965683 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:03:23.815759 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.815723 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm"] Oct 30 16:03:23.816243 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.816227 4401 cpu_manager.go:414] "containerMap: removing container" podUID="ba205c44-4823-4b48-b193-61b74f738f04" containerName="revoke-keys" Oct 30 16:03:23.816288 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.816244 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba205c44-4823-4b48-b193-61b74f738f04" containerName="revoke-keys" Oct 30 16:03:23.816325 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.816313 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba205c44-4823-4b48-b193-61b74f738f04" containerName="revoke-keys" Oct 30 16:03:23.821650 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.821632 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:23.839505 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.839480 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm"] Oct 30 16:03:23.839700 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.839680 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:03:23.849218 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.849184 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:03:23.849337 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.849190 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:03:23.853583 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.853566 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:03:23.978283 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.978246 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:23.978459 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.978311 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcsvb\" (UniqueName: \"kubernetes.io/projected/39ac591c-94da-464c-8bf1-a895223e78c2-kube-api-access-gcsvb\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:23.978459 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:23.978408 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.079171 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.079094 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcsvb\" (UniqueName: \"kubernetes.io/projected/39ac591c-94da-464c-8bf1-a895223e78c2-kube-api-access-gcsvb\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.079171 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.079147 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.079351 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.079228 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.079909 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.079880 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.081751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.081730 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.106622 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.106601 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcsvb\" (UniqueName: \"kubernetes.io/projected/39ac591c-94da-464c-8bf1-a895223e78c2-kube-api-access-gcsvb\") pod \"maas-api-revoke-sub-a4aef7d7-zkkzm\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.134130 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.134110 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:24.438470 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:24.438433 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm"] Oct 30 16:03:24.440588 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:03:24.440559 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ac591c_94da_464c_8bf1_a895223e78c2.slice/crio-70cf2416bb74eafe3eb51069f01ddfdb2dd7dd41efbe505f6c541891d9b950f0 WatchSource:0}: Error finding container 70cf2416bb74eafe3eb51069f01ddfdb2dd7dd41efbe505f6c541891d9b950f0: Status 404 returned error can't find the container with id 70cf2416bb74eafe3eb51069f01ddfdb2dd7dd41efbe505f6c541891d9b950f0 Oct 30 16:03:25.078982 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:25.078896 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" event={"ID":"39ac591c-94da-464c-8bf1-a895223e78c2","Type":"ContainerStarted","Data":"e9d6770b3480f425edeb32c2c29e10d4f570ecf7c0123a596baa3c604fcadf17"} Oct 30 16:03:25.078982 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:25.078977 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" event={"ID":"39ac591c-94da-464c-8bf1-a895223e78c2","Type":"ContainerStarted","Data":"70cf2416bb74eafe3eb51069f01ddfdb2dd7dd41efbe505f6c541891d9b950f0"} Oct 30 16:03:25.131584 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:25.131535 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" podStartSLOduration=2.131522427 podStartE2EDuration="2.131522427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:03:25.128603604 +0000 UTC m=+1513.827876761" watchObservedRunningTime="2026-10-30 16:03:25.131522427 +0000 UTC m=+1513.830795634" Oct 30 16:03:26.086614 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:26.086581 4401 generic.go:356] "Generic (PLEG): container finished" podID="39ac591c-94da-464c-8bf1-a895223e78c2" containerID="e9d6770b3480f425edeb32c2c29e10d4f570ecf7c0123a596baa3c604fcadf17" exitCode=0 Oct 30 16:03:26.087040 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:26.086673 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" event={"ID":"39ac591c-94da-464c-8bf1-a895223e78c2","Type":"ContainerDied","Data":"e9d6770b3480f425edeb32c2c29e10d4f570ecf7c0123a596baa3c604fcadf17"} Oct 30 16:03:27.233606 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.233579 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:27.310797 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.310762 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ac591c-94da-464c-8bf1-a895223e78c2-kube-api-access-gcsvb\" (UniqueName: \"kubernetes.io/projected/39ac591c-94da-464c-8bf1-a895223e78c2-kube-api-access-gcsvb\") pod \"39ac591c-94da-464c-8bf1-a895223e78c2\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " Oct 30 16:03:27.311011 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.310888 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-xks-ca\") pod \"39ac591c-94da-464c-8bf1-a895223e78c2\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " Oct 30 16:03:27.311011 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.310989 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-openshift-ca\") pod \"39ac591c-94da-464c-8bf1-a895223e78c2\" (UID: \"39ac591c-94da-464c-8bf1-a895223e78c2\") " Oct 30 16:03:27.311409 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.311383 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-openshift-ca" pod "39ac591c-94da-464c-8bf1-a895223e78c2" (UID: "39ac591c-94da-464c-8bf1-a895223e78c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:03:27.312986 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.312958 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-xks-ca" pod "39ac591c-94da-464c-8bf1-a895223e78c2" (UID: "39ac591c-94da-464c-8bf1-a895223e78c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:03:27.313115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.313054 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ac591c-94da-464c-8bf1-a895223e78c2-kube-api-access-gcsvb" pod "39ac591c-94da-464c-8bf1-a895223e78c2" (UID: "39ac591c-94da-464c-8bf1-a895223e78c2"). InnerVolumeSpecName "kube-api-access-gcsvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:03:27.412131 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.412100 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:27.412131 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.412127 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcsvb\" (UniqueName: \"kubernetes.io/projected/39ac591c-94da-464c-8bf1-a895223e78c2-kube-api-access-gcsvb\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:27.412131 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:27.412139 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ac591c-94da-464c-8bf1-a895223e78c2-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:28.100949 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:28.100917 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" Oct 30 16:03:28.101124 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:28.100916 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm" event={"ID":"39ac591c-94da-464c-8bf1-a895223e78c2","Type":"ContainerDied","Data":"70cf2416bb74eafe3eb51069f01ddfdb2dd7dd41efbe505f6c541891d9b950f0"} Oct 30 16:03:28.101124 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:28.101047 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70cf2416bb74eafe3eb51069f01ddfdb2dd7dd41efbe505f6c541891d9b950f0" Oct 30 16:03:32.032874 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:32.032817 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg"] Oct 30 16:03:32.048900 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:32.048861 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1fd93a5-wmfhg"] Oct 30 16:03:33.426609 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:33.426575 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc868aa-311e-400f-bf44-808911468167" path="/var/lib/kubelet/pods/6fc868aa-311e-400f-bf44-808911468167/volumes" Oct 30 16:03:35.041082 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:35.041044 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs"] Oct 30 16:03:35.051621 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:35.051592 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fd258fb-zr5xs"] Oct 30 16:03:35.283206 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:35.283168 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:03:35.427419 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:35.427390 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b71dd18-7858-44b4-ad75-9d57e2f8c90e" path="/var/lib/kubelet/pods/7b71dd18-7858-44b4-ad75-9d57e2f8c90e/volumes" Oct 30 16:03:37.030559 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:37.030526 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh"] Oct 30 16:03:37.044635 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:37.044606 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48a7e5a1-g5srh"] Oct 30 16:03:37.434179 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:37.434141 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cf706a2-8886-4104-bd12-b13608ae8cbd" path="/var/lib/kubelet/pods/7cf706a2-8886-4104-bd12-b13608ae8cbd/volumes" Oct 30 16:03:39.787595 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:39.787561 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:03:44.955283 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.955250 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb"] Oct 30 16:03:44.955731 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.955719 4401 cpu_manager.go:414] "containerMap: removing container" podUID="39ac591c-94da-464c-8bf1-a895223e78c2" containerName="revoke-keys" Oct 30 16:03:44.955783 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.955732 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ac591c-94da-464c-8bf1-a895223e78c2" containerName="revoke-keys" Oct 30 16:03:44.955853 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.955794 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ac591c-94da-464c-8bf1-a895223e78c2" containerName="revoke-keys" Oct 30 16:03:44.960206 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.960189 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:44.976972 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.976948 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb"] Oct 30 16:03:44.986856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.986818 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:03:44.986978 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.986963 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:03:44.987036 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.987024 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:03:44.990779 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:44.990764 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:03:45.053397 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.053365 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlqr\" (UniqueName: \"kubernetes.io/projected/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-kube-api-access-5mlqr\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.053673 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.053437 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.053673 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.053478 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.154043 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.154011 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlqr\" (UniqueName: \"kubernetes.io/projected/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-kube-api-access-5mlqr\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.154210 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.154071 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.154210 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.154099 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.154868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.154839 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.156562 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.156538 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.179769 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.179740 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlqr\" (UniqueName: \"kubernetes.io/projected/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-kube-api-access-5mlqr\") pod \"maas-api-revoke-sub-79805ca9-mv7jb\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.273149 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.273078 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:45.576857 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:03:45.576799 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc3e36_c0f5_4012_a058_eee8f23eab7b.slice/crio-330aaa129b9eca7628c07e8a6257ec221f918871fcec9a04529747ad70976d2c WatchSource:0}: Error finding container 330aaa129b9eca7628c07e8a6257ec221f918871fcec9a04529747ad70976d2c: Status 404 returned error can't find the container with id 330aaa129b9eca7628c07e8a6257ec221f918871fcec9a04529747ad70976d2c Oct 30 16:03:45.576857 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:45.576837 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb"] Oct 30 16:03:46.292267 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:46.292183 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" event={"ID":"2bcc3e36-c0f5-4012-a058-eee8f23eab7b","Type":"ContainerStarted","Data":"63c38758b4dd16d423f00d261a83f50edadc1d9525c0cdea1a6934dee1231122"} Oct 30 16:03:46.292267 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:46.292243 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" event={"ID":"2bcc3e36-c0f5-4012-a058-eee8f23eab7b","Type":"ContainerStarted","Data":"330aaa129b9eca7628c07e8a6257ec221f918871fcec9a04529747ad70976d2c"} Oct 30 16:03:46.339219 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:46.339177 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" podStartSLOduration=2.339164035 podStartE2EDuration="2.339164035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:03:46.337853587 +0000 UTC m=+1535.037126738" watchObservedRunningTime="2026-10-30 16:03:46.339164035 +0000 UTC m=+1535.038437206" Oct 30 16:03:47.300578 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:47.300541 4401 generic.go:356] "Generic (PLEG): container finished" podID="2bcc3e36-c0f5-4012-a058-eee8f23eab7b" containerID="63c38758b4dd16d423f00d261a83f50edadc1d9525c0cdea1a6934dee1231122" exitCode=0 Oct 30 16:03:47.300986 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:47.300626 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" event={"ID":"2bcc3e36-c0f5-4012-a058-eee8f23eab7b","Type":"ContainerDied","Data":"63c38758b4dd16d423f00d261a83f50edadc1d9525c0cdea1a6934dee1231122"} Oct 30 16:03:48.452164 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.452135 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:48.481524 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.481491 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-xks-ca\") pod \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " Oct 30 16:03:48.481694 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.481538 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-openshift-ca\") pod \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " Oct 30 16:03:48.481694 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.481626 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-kube-api-access-5mlqr\" (UniqueName: \"kubernetes.io/projected/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-kube-api-access-5mlqr\") pod \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\" (UID: \"2bcc3e36-c0f5-4012-a058-eee8f23eab7b\") " Oct 30 16:03:48.482024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.481993 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-openshift-ca" pod "2bcc3e36-c0f5-4012-a058-eee8f23eab7b" (UID: "2bcc3e36-c0f5-4012-a058-eee8f23eab7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:03:48.483697 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.483674 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-xks-ca" pod "2bcc3e36-c0f5-4012-a058-eee8f23eab7b" (UID: "2bcc3e36-c0f5-4012-a058-eee8f23eab7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:03:48.483815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.483733 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-kube-api-access-5mlqr" pod "2bcc3e36-c0f5-4012-a058-eee8f23eab7b" (UID: "2bcc3e36-c0f5-4012-a058-eee8f23eab7b"). InnerVolumeSpecName "kube-api-access-5mlqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:03:48.582456 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.582371 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:48.582456 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.582404 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:48.582456 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:48.582414 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mlqr\" (UniqueName: \"kubernetes.io/projected/2bcc3e36-c0f5-4012-a058-eee8f23eab7b-kube-api-access-5mlqr\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:03:49.315256 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:49.315221 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" Oct 30 16:03:49.315434 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:49.315224 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb" event={"ID":"2bcc3e36-c0f5-4012-a058-eee8f23eab7b","Type":"ContainerDied","Data":"330aaa129b9eca7628c07e8a6257ec221f918871fcec9a04529747ad70976d2c"} Oct 30 16:03:49.315434 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:49.315349 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="330aaa129b9eca7628c07e8a6257ec221f918871fcec9a04529747ad70976d2c" Oct 30 16:03:51.966397 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:03:51.966356 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:03:57.020069 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:03:57.020035 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:04:01.035388 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.035347 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk"] Oct 30 16:04:01.049391 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.049358 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194f8cfb-dnqwk"] Oct 30 16:04:01.427296 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.427261 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f31ffe21-4522-4e15-bb9e-e9e0bfe13225" path="/var/lib/kubelet/pods/f31ffe21-4522-4e15-bb9e-e9e0bfe13225/volumes" Oct 30 16:04:01.889307 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.889268 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485"] Oct 30 16:04:01.889761 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.889747 4401 cpu_manager.go:414] "containerMap: removing container" podUID="2bcc3e36-c0f5-4012-a058-eee8f23eab7b" containerName="revoke-keys" Oct 30 16:04:01.889809 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.889762 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcc3e36-c0f5-4012-a058-eee8f23eab7b" containerName="revoke-keys" Oct 30 16:04:01.889894 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.889857 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcc3e36-c0f5-4012-a058-eee8f23eab7b" containerName="revoke-keys" Oct 30 16:04:01.894386 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.894368 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:01.915089 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.915064 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:04:01.929056 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.929027 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:04:01.929754 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.929737 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:04:01.929844 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.929794 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:04:01.932884 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.932863 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485"] Oct 30 16:04:01.957983 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.957953 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:01.958132 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.958111 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:01.958180 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:01.958140 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4bzs\" (UniqueName: \"kubernetes.io/projected/d7931d90-a646-44df-94e3-70efb20efad0-kube-api-access-r4bzs\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.037756 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.037717 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx"] Oct 30 16:04:02.040805 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.040777 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6"] Oct 30 16:04:02.043285 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.043258 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c878956b-scznx"] Oct 30 16:04:02.045712 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.045691 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae73c45b-l9wk6"] Oct 30 16:04:02.059156 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.059128 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.059246 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.059159 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4bzs\" (UniqueName: \"kubernetes.io/projected/d7931d90-a646-44df-94e3-70efb20efad0-kube-api-access-r4bzs\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.059246 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.059223 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.059774 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.059753 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.061495 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.061477 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.092066 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.092038 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4bzs\" (UniqueName: \"kubernetes.io/projected/d7931d90-a646-44df-94e3-70efb20efad0-kube-api-access-r4bzs\") pod \"maas-api-revoke-sub-0b7e8edb-2w485\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.207010 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.206929 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:02.520610 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:02.520566 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485"] Oct 30 16:04:02.520919 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:04:02.520894 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7931d90_a646_44df_94e3_70efb20efad0.slice/crio-49ce03d9ba954e5c0ab7ed7eab421f5a00e8d42c3c54a566f5e719ec00b578b6 WatchSource:0}: Error finding container 49ce03d9ba954e5c0ab7ed7eab421f5a00e8d42c3c54a566f5e719ec00b578b6: Status 404 returned error can't find the container with id 49ce03d9ba954e5c0ab7ed7eab421f5a00e8d42c3c54a566f5e719ec00b578b6 Oct 30 16:04:03.428494 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:03.428461 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da206b1-06a1-44fa-9a61-29211b75cc57" path="/var/lib/kubelet/pods/9da206b1-06a1-44fa-9a61-29211b75cc57/volumes" Oct 30 16:04:03.428986 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:03.428967 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675" path="/var/lib/kubelet/pods/9ddbd7c6-7d5a-4482-ba7b-8cd4f7010675/volumes" Oct 30 16:04:03.429964 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:03.429942 4401 generic.go:356] "Generic (PLEG): container finished" podID="d7931d90-a646-44df-94e3-70efb20efad0" containerID="88004361ec4acd79fcb893eb50e31c52e8b041ad4a65a28f11029ec88ea1a874" exitCode=0 Oct 30 16:04:03.430041 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:03.430004 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" event={"ID":"d7931d90-a646-44df-94e3-70efb20efad0","Type":"ContainerDied","Data":"88004361ec4acd79fcb893eb50e31c52e8b041ad4a65a28f11029ec88ea1a874"} Oct 30 16:04:03.430041 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:03.430037 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" event={"ID":"d7931d90-a646-44df-94e3-70efb20efad0","Type":"ContainerStarted","Data":"49ce03d9ba954e5c0ab7ed7eab421f5a00e8d42c3c54a566f5e719ec00b578b6"} Oct 30 16:04:04.574531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.574504 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:04.685723 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.685687 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7931d90-a646-44df-94e3-70efb20efad0-kube-api-access-r4bzs\" (UniqueName: \"kubernetes.io/projected/d7931d90-a646-44df-94e3-70efb20efad0-kube-api-access-r4bzs\") pod \"d7931d90-a646-44df-94e3-70efb20efad0\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " Oct 30 16:04:04.685943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.685747 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-openshift-ca\") pod \"d7931d90-a646-44df-94e3-70efb20efad0\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " Oct 30 16:04:04.685943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.685885 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-xks-ca\") pod \"d7931d90-a646-44df-94e3-70efb20efad0\" (UID: \"d7931d90-a646-44df-94e3-70efb20efad0\") " Oct 30 16:04:04.686182 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.686155 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-openshift-ca" pod "d7931d90-a646-44df-94e3-70efb20efad0" (UID: "d7931d90-a646-44df-94e3-70efb20efad0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:04:04.686272 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.686254 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:04.688003 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.687943 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-xks-ca" pod "d7931d90-a646-44df-94e3-70efb20efad0" (UID: "d7931d90-a646-44df-94e3-70efb20efad0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:04:04.688003 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.687961 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7931d90-a646-44df-94e3-70efb20efad0-kube-api-access-r4bzs" pod "d7931d90-a646-44df-94e3-70efb20efad0" (UID: "d7931d90-a646-44df-94e3-70efb20efad0"). InnerVolumeSpecName "kube-api-access-r4bzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:04:04.786917 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.786882 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4bzs\" (UniqueName: \"kubernetes.io/projected/d7931d90-a646-44df-94e3-70efb20efad0-kube-api-access-r4bzs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:04.786917 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:04.786916 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7931d90-a646-44df-94e3-70efb20efad0-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:05.444008 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:05.443982 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" Oct 30 16:04:05.444185 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:05.444025 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485" event={"ID":"d7931d90-a646-44df-94e3-70efb20efad0","Type":"ContainerDied","Data":"49ce03d9ba954e5c0ab7ed7eab421f5a00e8d42c3c54a566f5e719ec00b578b6"} Oct 30 16:04:05.444185 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:05.444081 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49ce03d9ba954e5c0ab7ed7eab421f5a00e8d42c3c54a566f5e719ec00b578b6" Oct 30 16:04:12.680354 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.680321 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62"] Oct 30 16:04:12.681412 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.681022 4401 cpu_manager.go:414] "containerMap: removing container" podUID="d7931d90-a646-44df-94e3-70efb20efad0" containerName="revoke-keys" Oct 30 16:04:12.681412 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.681045 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7931d90-a646-44df-94e3-70efb20efad0" containerName="revoke-keys" Oct 30 16:04:12.681412 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.681191 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7931d90-a646-44df-94e3-70efb20efad0" containerName="revoke-keys" Oct 30 16:04:12.684734 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.684710 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.703671 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.703636 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62"] Oct 30 16:04:12.712901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.712865 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:04:12.712901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.712890 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:04:12.715247 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.715228 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:04:12.715453 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.715436 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:04:12.767612 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.767584 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwbp8\" (UniqueName: \"kubernetes.io/projected/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-kube-api-access-cwbp8\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.767759 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.767655 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.767857 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.767811 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.800524 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.800497 4401 scope.go:122] "RemoveContainer" containerID="bf4e08c283b7905f33c8a9661939eabc9eac5e7e1634950af98b1d3ad433ba11" Oct 30 16:04:12.809886 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.809866 4401 scope.go:122] "RemoveContainer" containerID="af965f06a21cc052c0f18b3bfb07e8d7f1fa6eb0f9f72f5882287ef579197a48" Oct 30 16:04:12.818605 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.818585 4401 scope.go:122] "RemoveContainer" containerID="ddeb44b23e5ac983a541798d7d876ab5f80910dc41de889504c605005e80d931" Oct 30 16:04:12.827339 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.827317 4401 scope.go:122] "RemoveContainer" containerID="92970131543bb6906028954a1e151d58c0593a483bdf92c79864cbdf116eab28" Oct 30 16:04:12.835868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.835847 4401 scope.go:122] "RemoveContainer" containerID="66a4889cafb2606610dc69bae2db186f00e7cd6a888e3fb3d440625822595b81" Oct 30 16:04:12.844414 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.844391 4401 scope.go:122] "RemoveContainer" containerID="34f9e593e3e7daa702a16320c63863c487bca266cd5662541a9694f91492b22c" Oct 30 16:04:12.852512 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.852492 4401 scope.go:122] "RemoveContainer" containerID="588a59429711851209df5731b74d2b2c3fcca9b726cdf9bc7f39cbfeaaf27540" Oct 30 16:04:12.860316 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.860299 4401 scope.go:122] "RemoveContainer" containerID="b29d431e87942d465caab007ed11e00ccc2153840594bb52f56bc4e9b95682f2" Oct 30 16:04:12.868410 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.868385 4401 scope.go:122] "RemoveContainer" containerID="ba25de8e19066d90e73b11558bee8a7d2fe912e9f64b2ee3550a0a4f368c2f11" Oct 30 16:04:12.868594 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.868575 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.868744 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.868720 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwbp8\" (UniqueName: \"kubernetes.io/projected/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-kube-api-access-cwbp8\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.868832 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.868797 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.869432 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.869413 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.870909 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.870884 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.883780 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.883757 4401 scope.go:122] "RemoveContainer" containerID="03b7e97207d6beedfca8b6dbbe4dec5bdade941bec9679a39af4712a30ef89ee" Oct 30 16:04:12.892301 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.892278 4401 scope.go:122] "RemoveContainer" containerID="17c47db85daf615598b61c550b9a26e0f628226c4c996769d9bfad6816579d47" Oct 30 16:04:12.894964 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.894940 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwbp8\" (UniqueName: \"kubernetes.io/projected/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-kube-api-access-cwbp8\") pod \"maas-api-revoke-sub-3d4156f1-4rl62\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:12.907777 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.907757 4401 scope.go:122] "RemoveContainer" containerID="60eb9f5049d0a0fcd63a6e63e6d2e8d79974fb139cc0b5d95179b85e3c5a545e" Oct 30 16:04:12.916538 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.916519 4401 scope.go:122] "RemoveContainer" containerID="8bfe222badf6fc046d2c0aa72f027ad3a4160ef8eef4633f28603eb0c582945c" Oct 30 16:04:12.998285 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:12.998260 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:13.213799 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:13.213715 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:04:13.302702 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:13.302667 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62"] Oct 30 16:04:13.303549 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:04:13.303504 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ab29a7_3cbd_4c99_92c5_a3cd837d1e28.slice/crio-c5810ab2c0bc23470752f8343ff6bdbea2102a771eda055957fd800cebb6589e WatchSource:0}: Error finding container c5810ab2c0bc23470752f8343ff6bdbea2102a771eda055957fd800cebb6589e: Status 404 returned error can't find the container with id c5810ab2c0bc23470752f8343ff6bdbea2102a771eda055957fd800cebb6589e Oct 30 16:04:13.498148 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:13.498116 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" event={"ID":"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28","Type":"ContainerStarted","Data":"c5810ab2c0bc23470752f8343ff6bdbea2102a771eda055957fd800cebb6589e"} Oct 30 16:04:14.506368 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:14.506327 4401 generic.go:356] "Generic (PLEG): container finished" podID="62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" containerID="8de5f1a36fab89a0e8e5d068b129b65b4359a729665eea71f8a00fcf3007b486" exitCode=0 Oct 30 16:04:14.506789 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:14.506396 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" event={"ID":"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28","Type":"ContainerDied","Data":"8de5f1a36fab89a0e8e5d068b129b65b4359a729665eea71f8a00fcf3007b486"} Oct 30 16:04:15.653598 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.653573 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:15.796201 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.796117 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-kube-api-access-cwbp8\" (UniqueName: \"kubernetes.io/projected/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-kube-api-access-cwbp8\") pod \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " Oct 30 16:04:15.796201 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.796174 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-xks-ca\") pod \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " Oct 30 16:04:15.796394 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.796361 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-openshift-ca\") pod \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\" (UID: \"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28\") " Oct 30 16:04:15.796727 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.796695 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-openshift-ca" pod "62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" (UID: "62ab29a7-3cbd-4c99-92c5-a3cd837d1e28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:04:15.796882 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.796801 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:15.798178 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.798145 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-xks-ca" pod "62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" (UID: "62ab29a7-3cbd-4c99-92c5-a3cd837d1e28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:04:15.798266 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.798175 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-kube-api-access-cwbp8" pod "62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" (UID: "62ab29a7-3cbd-4c99-92c5-a3cd837d1e28"). InnerVolumeSpecName "kube-api-access-cwbp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:04:15.898144 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.898109 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwbp8\" (UniqueName: \"kubernetes.io/projected/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-kube-api-access-cwbp8\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:15.898144 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:15.898137 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:16.520272 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:16.520240 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" Oct 30 16:04:16.520272 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:16.520263 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62" event={"ID":"62ab29a7-3cbd-4c99-92c5-a3cd837d1e28","Type":"ContainerDied","Data":"c5810ab2c0bc23470752f8343ff6bdbea2102a771eda055957fd800cebb6589e"} Oct 30 16:04:16.520495 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:16.520306 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5810ab2c0bc23470752f8343ff6bdbea2102a771eda055957fd800cebb6589e" Oct 30 16:04:21.966686 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:04:21.966645 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:04:23.973860 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:23.973804 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:04:28.034001 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:28.033962 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc"] Oct 30 16:04:28.047202 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:28.047173 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8325bc7d-l88sc"] Oct 30 16:04:29.300189 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:29.300152 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:04:29.426850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:29.426805 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5420580-a0a7-4211-be18-7c45a354508d" path="/var/lib/kubelet/pods/e5420580-a0a7-4211-be18-7c45a354508d/volumes" Oct 30 16:04:34.019627 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.019588 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62"] Oct 30 16:04:34.033445 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.033417 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4rl62"] Oct 30 16:04:34.071954 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.071921 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8"] Oct 30 16:04:34.072394 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.072380 4401 cpu_manager.go:414] "containerMap: removing container" podUID="62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" containerName="revoke-keys" Oct 30 16:04:34.072437 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.072396 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" containerName="revoke-keys" Oct 30 16:04:34.072510 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.072498 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" containerName="revoke-keys" Oct 30 16:04:34.075745 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.075726 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.092515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.092479 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:04:34.092701 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.092562 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:04:34.092701 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.092585 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:04:34.092701 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.092637 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:04:34.094715 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.094685 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8"] Oct 30 16:04:34.119358 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.119324 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.119592 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.119376 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48pv7\" (UniqueName: \"kubernetes.io/projected/23f5f9d0-3143-4883-988f-46e6a0483985-kube-api-access-48pv7\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.119901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.119870 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.220446 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.220402 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.220642 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.220478 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.220642 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.220612 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48pv7\" (UniqueName: \"kubernetes.io/projected/23f5f9d0-3143-4883-988f-46e6a0483985-kube-api-access-48pv7\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.221230 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.221211 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.222793 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.222774 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.246071 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.246036 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48pv7\" (UniqueName: \"kubernetes.io/projected/23f5f9d0-3143-4883-988f-46e6a0483985-kube-api-access-48pv7\") pod \"maas-api-revoke-sub-3d4156f1-xltv8\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.388759 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.388723 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:34.690631 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:34.690600 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8"] Oct 30 16:04:34.691367 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:04:34.691338 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f5f9d0_3143_4883_988f_46e6a0483985.slice/crio-2e801afd563cbe982f3fed231c42c89f479797083e248d7075d2a65eac8e477c WatchSource:0}: Error finding container 2e801afd563cbe982f3fed231c42c89f479797083e248d7075d2a65eac8e477c: Status 404 returned error can't find the container with id 2e801afd563cbe982f3fed231c42c89f479797083e248d7075d2a65eac8e477c Oct 30 16:04:35.427713 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:35.427677 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ab29a7-3cbd-4c99-92c5-a3cd837d1e28" path="/var/lib/kubelet/pods/62ab29a7-3cbd-4c99-92c5-a3cd837d1e28/volumes" Oct 30 16:04:35.696114 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:35.696029 4401 generic.go:356] "Generic (PLEG): container finished" podID="23f5f9d0-3143-4883-988f-46e6a0483985" containerID="73b93689b6d57d806d05934e47d2ab70f492dc099df7a74f256c2a5e53b4eb21" exitCode=0 Oct 30 16:04:35.696268 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:35.696114 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" event={"ID":"23f5f9d0-3143-4883-988f-46e6a0483985","Type":"ContainerDied","Data":"73b93689b6d57d806d05934e47d2ab70f492dc099df7a74f256c2a5e53b4eb21"} Oct 30 16:04:35.696268 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:35.696172 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" event={"ID":"23f5f9d0-3143-4883-988f-46e6a0483985","Type":"ContainerStarted","Data":"2e801afd563cbe982f3fed231c42c89f479797083e248d7075d2a65eac8e477c"} Oct 30 16:04:36.836665 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.836640 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:36.946613 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.946583 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f5f9d0-3143-4883-988f-46e6a0483985-kube-api-access-48pv7\" (UniqueName: \"kubernetes.io/projected/23f5f9d0-3143-4883-988f-46e6a0483985-kube-api-access-48pv7\") pod \"23f5f9d0-3143-4883-988f-46e6a0483985\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " Oct 30 16:04:36.946757 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.946619 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-openshift-ca\") pod \"23f5f9d0-3143-4883-988f-46e6a0483985\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " Oct 30 16:04:36.946879 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.946860 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-xks-ca\") pod \"23f5f9d0-3143-4883-988f-46e6a0483985\" (UID: \"23f5f9d0-3143-4883-988f-46e6a0483985\") " Oct 30 16:04:36.947077 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.947057 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-openshift-ca" pod "23f5f9d0-3143-4883-988f-46e6a0483985" (UID: "23f5f9d0-3143-4883-988f-46e6a0483985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:04:36.947226 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.947206 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:36.948765 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.948739 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f5f9d0-3143-4883-988f-46e6a0483985-kube-api-access-48pv7" pod "23f5f9d0-3143-4883-988f-46e6a0483985" (UID: "23f5f9d0-3143-4883-988f-46e6a0483985"). InnerVolumeSpecName "kube-api-access-48pv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:04:36.948765 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:36.948745 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-xks-ca" pod "23f5f9d0-3143-4883-988f-46e6a0483985" (UID: "23f5f9d0-3143-4883-988f-46e6a0483985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:04:37.048077 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:37.048047 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23f5f9d0-3143-4883-988f-46e6a0483985-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:37.048077 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:37.048075 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48pv7\" (UniqueName: \"kubernetes.io/projected/23f5f9d0-3143-4883-988f-46e6a0483985-kube-api-access-48pv7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:04:37.709295 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:37.709216 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" event={"ID":"23f5f9d0-3143-4883-988f-46e6a0483985","Type":"ContainerDied","Data":"2e801afd563cbe982f3fed231c42c89f479797083e248d7075d2a65eac8e477c"} Oct 30 16:04:37.709295 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:37.709268 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e801afd563cbe982f3fed231c42c89f479797083e248d7075d2a65eac8e477c" Oct 30 16:04:37.709295 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:37.709225 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8" Oct 30 16:04:46.184365 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:46.184329 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:04:51.044790 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:51.044749 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t"] Oct 30 16:04:51.058669 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:51.058638 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a89e2376-jx47t"] Oct 30 16:04:51.426927 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:51.426891 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28f6e172-6f85-4f93-ba69-f9cf055dc901" path="/var/lib/kubelet/pods/28f6e172-6f85-4f93-ba69-f9cf055dc901/volumes" Oct 30 16:04:51.504924 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:51.504886 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:04:51.966379 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:04:51.966347 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:04:57.827772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:57.827728 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8"] Oct 30 16:04:57.841590 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:57.841562 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xltv8"] Oct 30 16:04:59.427053 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:04:59.427023 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23f5f9d0-3143-4883-988f-46e6a0483985" path="/var/lib/kubelet/pods/23f5f9d0-3143-4883-988f-46e6a0483985/volumes" Oct 30 16:05:09.997143 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:09.997113 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:05:12.994865 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:12.994812 4401 scope.go:122] "RemoveContainer" containerID="29e11605a83b6839fc133c556f895ed6ee025a174f1eb25cbf6ff2c8f70f0bc5" Oct 30 16:05:13.004134 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:13.004114 4401 scope.go:122] "RemoveContainer" containerID="a5fcce886fbbc66bcc932edf908f18e12060472536b00b3c072b7c1474a7d641" Oct 30 16:05:13.012374 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:13.012352 4401 scope.go:122] "RemoveContainer" containerID="d56babca00563a2bb965b8a1bcef7b9fc2be779d1edfacf0645437868e33247a" Oct 30 16:05:13.020781 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:13.020760 4401 scope.go:122] "RemoveContainer" containerID="8481c9b0b892b5744fffaf44d018d87b7fd56976b423f12f5b91606f25ca8551" Oct 30 16:05:13.028875 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:13.028858 4401 scope.go:122] "RemoveContainer" containerID="42609da5c396872a3b78715e05bf2fffe7e1e6b3d1d8dafe53e4835e6b3081c2" Oct 30 16:05:13.039364 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:13.039336 4401 scope.go:122] "RemoveContainer" containerID="1eccffef68629524c9a8cdf65879c3d9397759c6f6cae5340012cbe854aff176" Oct 30 16:05:13.047663 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:13.047645 4401 scope.go:122] "RemoveContainer" containerID="bdf1680fcc9a7f04c8abfc7a0487c3912c0f0ee5df64c432fb3dff1680fd5fa2" Oct 30 16:05:13.055557 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:13.055539 4401 scope.go:122] "RemoveContainer" containerID="ed196a7ca06e2cc6f9a7453a825ed4d90d0c876e04b3db76d885e73e11503aa8" Oct 30 16:05:15.267685 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:15.267647 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:05:21.966674 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:05:21.966633 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:05:25.272006 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.271969 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp"] Oct 30 16:05:25.272473 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.272460 4401 cpu_manager.go:414] "containerMap: removing container" podUID="23f5f9d0-3143-4883-988f-46e6a0483985" containerName="revoke-keys" Oct 30 16:05:25.272518 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.272474 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f5f9d0-3143-4883-988f-46e6a0483985" containerName="revoke-keys" Oct 30 16:05:25.272559 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.272542 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f5f9d0-3143-4883-988f-46e6a0483985" containerName="revoke-keys" Oct 30 16:05:25.278225 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.278199 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.295570 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.295536 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp"] Oct 30 16:05:25.298135 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.298110 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6a2c74\"" Oct 30 16:05:25.306321 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.306286 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0e9e570c-dockercfg-bjb7l\"" Oct 30 16:05:25.341323 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.341288 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr"] Oct 30 16:05:25.345988 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.345963 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.360764 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.360734 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr"] Oct 30 16:05:25.425588 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.425537 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916becdc-4e36-4eb5-b87d-6e1146d69acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.425952 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.425929 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/4c4e7c9c-fca4-4909-8bdc-1989807fd335-kube-api-access-mtf2j\") pod \"payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.426142 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.426110 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4e7c9c-fca4-4909-8bdc-1989807fd335-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.426337 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.426321 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5c4t\" (UniqueName: \"kubernetes.io/projected/916becdc-4e36-4eb5-b87d-6e1146d69acd-kube-api-access-j5c4t\") pod \"payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.527353 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.527274 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916becdc-4e36-4eb5-b87d-6e1146d69acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.527353 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.527307 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/4c4e7c9c-fca4-4909-8bdc-1989807fd335-kube-api-access-mtf2j\") pod \"payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.527353 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.527324 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4e7c9c-fca4-4909-8bdc-1989807fd335-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.527651 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.527365 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5c4t\" (UniqueName: \"kubernetes.io/projected/916becdc-4e36-4eb5-b87d-6e1146d69acd-kube-api-access-j5c4t\") pod \"payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.528099 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.528079 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4e7c9c-fca4-4909-8bdc-1989807fd335-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.528249 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.528230 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916becdc-4e36-4eb5-b87d-6e1146d69acd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.552976 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.552941 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/4c4e7c9c-fca4-4909-8bdc-1989807fd335-kube-api-access-mtf2j\") pod \"payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.553112 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.552941 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5c4t\" (UniqueName: \"kubernetes.io/projected/916becdc-4e36-4eb5-b87d-6e1146d69acd-kube-api-access-j5c4t\") pod \"payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.591960 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.591936 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:25.663971 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.662064 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:25.899144 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.899116 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp"] Oct 30 16:05:25.900289 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:05:25.900263 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4e7c9c_fca4_4909_8bdc_1989807fd335.slice/crio-7791aab6e0fb55c58f92085a1b482a1dd0eb453cca3fea08d7afaa5f8435f0ac WatchSource:0}: Error finding container 7791aab6e0fb55c58f92085a1b482a1dd0eb453cca3fea08d7afaa5f8435f0ac: Status 404 returned error can't find the container with id 7791aab6e0fb55c58f92085a1b482a1dd0eb453cca3fea08d7afaa5f8435f0ac Oct 30 16:05:25.975290 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:25.975262 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr"] Oct 30 16:05:25.975424 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:05:25.975403 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916becdc_4e36_4eb5_b87d_6e1146d69acd.slice/crio-88bf1957e895fc32d697b6760dfd6e3d8c6ca3c893aedd383f5bb93a1c289d7f WatchSource:0}: Error finding container 88bf1957e895fc32d697b6760dfd6e3d8c6ca3c893aedd383f5bb93a1c289d7f: Status 404 returned error can't find the container with id 88bf1957e895fc32d697b6760dfd6e3d8c6ca3c893aedd383f5bb93a1c289d7f Oct 30 16:05:26.005334 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:26.005305 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" event={"ID":"916becdc-4e36-4eb5-b87d-6e1146d69acd","Type":"ContainerStarted","Data":"88bf1957e895fc32d697b6760dfd6e3d8c6ca3c893aedd383f5bb93a1c289d7f"} Oct 30 16:05:26.006896 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:26.006870 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" event={"ID":"4c4e7c9c-fca4-4909-8bdc-1989807fd335","Type":"ContainerStarted","Data":"3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230"} Oct 30 16:05:26.007015 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:26.006919 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" event={"ID":"4c4e7c9c-fca4-4909-8bdc-1989807fd335","Type":"ContainerStarted","Data":"7791aab6e0fb55c58f92085a1b482a1dd0eb453cca3fea08d7afaa5f8435f0ac"} Oct 30 16:05:26.007015 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:26.006964 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:26.051862 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:26.051798 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" podStartSLOduration=1.051783492 podStartE2EDuration="1.051783492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:05:26.050407631 +0000 UTC m=+1634.749680785" watchObservedRunningTime="2026-10-30 16:05:26.051783492 +0000 UTC m=+1634.751056695" Oct 30 16:05:27.014046 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:27.014009 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" event={"ID":"916becdc-4e36-4eb5-b87d-6e1146d69acd","Type":"ContainerStarted","Data":"67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75"} Oct 30 16:05:27.014489 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:27.014183 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:27.057783 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:27.057733 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" podStartSLOduration=2.05771983 podStartE2EDuration="2.05771983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:05:27.057162628 +0000 UTC m=+1635.756435785" watchObservedRunningTime="2026-10-30 16:05:27.05771983 +0000 UTC m=+1635.756992997" Oct 30 16:05:28.891738 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:28.891698 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:05:37.018015 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:37.017986 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:05:38.023163 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:38.023134 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:05:41.326778 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.326737 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt"] Oct 30 16:05:41.331046 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.331025 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.349295 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.349269 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt"] Oct 30 16:05:41.360022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.359987 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6a2c74\"/\"kube-root-ca.crt\"" Oct 30 16:05:41.360146 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.360027 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6a2c74\"/\"openshift-service-ca.crt\"" Oct 30 16:05:41.364643 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.364624 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6a2c74\"/\"simulator-single-main-6a2c74-kserve-self-signed-certs\"" Oct 30 16:05:41.364643 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.364624 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6a2c74\"/\"default-dockercfg-qwz97\"" Oct 30 16:05:41.492692 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.492661 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-dshm\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.492692 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.492696 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zkhf\" (UniqueName: \"kubernetes.io/projected/1a988a3f-15a6-4334-be34-ff54a7327010-kube-api-access-7zkhf\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.492957 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.492724 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-tmp-dir\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.492957 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.492894 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-model-cache\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.492957 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.492926 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a988a3f-15a6-4334-be34-ff54a7327010-tls-certs\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.493076 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.493048 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-home\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.594224 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.594115 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-dshm\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.594224 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.594172 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zkhf\" (UniqueName: \"kubernetes.io/projected/1a988a3f-15a6-4334-be34-ff54a7327010-kube-api-access-7zkhf\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.594224 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.594222 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-tmp-dir\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.594533 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.594265 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-model-cache\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.594533 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.594298 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a988a3f-15a6-4334-be34-ff54a7327010-tls-certs\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.594533 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.594385 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-home\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.595480 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.595451 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-tmp-dir\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.595691 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.595663 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-model-cache\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.595790 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.595714 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-home\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.598929 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.598894 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a988a3f-15a6-4334-be34-ff54a7327010-tls-certs\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.599910 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.599886 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-dshm\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.622011 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.621990 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zkhf\" (UniqueName: \"kubernetes.io/projected/1a988a3f-15a6-4334-be34-ff54a7327010-kube-api-access-7zkhf\") pod \"simulator-single-main-6a2c74-kserve-78d6856799-sgmmt\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.644004 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.643980 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:41.952875 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:41.952853 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt"] Oct 30 16:05:41.953587 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:05:41.953565 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a988a3f_15a6_4334_be34_ff54a7327010.slice/crio-022eaed9cc8729a27ee43c92c67bb7d7b62293f5075de5981de8110b9f9c9cd9 WatchSource:0}: Error finding container 022eaed9cc8729a27ee43c92c67bb7d7b62293f5075de5981de8110b9f9c9cd9: Status 404 returned error can't find the container with id 022eaed9cc8729a27ee43c92c67bb7d7b62293f5075de5981de8110b9f9c9cd9 Oct 30 16:05:42.111699 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:42.111660 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" event={"ID":"1a988a3f-15a6-4334-be34-ff54a7327010","Type":"ContainerStarted","Data":"4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea"} Oct 30 16:05:42.111699 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:42.111702 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" event={"ID":"1a988a3f-15a6-4334-be34-ff54a7327010","Type":"ContainerStarted","Data":"022eaed9cc8729a27ee43c92c67bb7d7b62293f5075de5981de8110b9f9c9cd9"} Oct 30 16:05:42.162138 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:42.162084 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" podStartSLOduration=1.162069738 podStartE2EDuration="1.162069738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:05:42.159414063 +0000 UTC m=+1650.858687220" watchObservedRunningTime="2026-10-30 16:05:42.162069738 +0000 UTC m=+1650.861342943" Oct 30 16:05:42.644620 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:42.644582 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:44.042690 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:44.042655 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s"] Oct 30 16:05:44.055489 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:44.055459 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-qt79s"] Oct 30 16:05:44.058123 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:44.058099 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:05:45.427153 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:45.427119 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d1dc1b-f796-457a-9d2e-600d90fe51ac" path="/var/lib/kubelet/pods/55d1dc1b-f796-457a-9d2e-600d90fe51ac/volumes" Oct 30 16:05:51.644915 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:51.644874 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:51.657991 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:51.657967 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:51.966484 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:05:51.966399 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:05:52.182907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:52.182880 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:05:58.406769 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:05:58.406691 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:06:04.449237 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.449197 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb"] Oct 30 16:06:04.452733 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.452717 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.470537 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.470508 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb"] Oct 30 16:06:04.481390 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.481367 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:06:04.481390 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.481373 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:06:04.481543 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.481391 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:06:04.483150 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.483129 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:06:04.540774 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.540741 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dn7\" (UniqueName: \"kubernetes.io/projected/e2cd27cc-3d42-4560-820e-015a5acac66b-kube-api-access-26dn7\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.540916 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.540888 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.541005 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.540954 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.643656 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.643607 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.644006 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.643973 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dn7\" (UniqueName: \"kubernetes.io/projected/e2cd27cc-3d42-4560-820e-015a5acac66b-kube-api-access-26dn7\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.644283 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.644109 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.645564 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.645538 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.652506 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.652477 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.670950 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.670917 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dn7\" (UniqueName: \"kubernetes.io/projected/e2cd27cc-3d42-4560-820e-015a5acac66b-kube-api-access-26dn7\") pod \"maas-api-revoke-sub-6c5182ff-jqxrb\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:04.766601 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:04.766525 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:05.072331 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:05.072295 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb"] Oct 30 16:06:05.072548 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:06:05.072523 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2cd27cc_3d42_4560_820e_015a5acac66b.slice/crio-b0d19e76b9b7582e1b5ca830902dbc3a633baab9edde29c91602fa8cf34cbb48 WatchSource:0}: Error finding container b0d19e76b9b7582e1b5ca830902dbc3a633baab9edde29c91602fa8cf34cbb48: Status 404 returned error can't find the container with id b0d19e76b9b7582e1b5ca830902dbc3a633baab9edde29c91602fa8cf34cbb48 Oct 30 16:06:05.253258 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:05.253218 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" event={"ID":"e2cd27cc-3d42-4560-820e-015a5acac66b","Type":"ContainerStarted","Data":"b0d19e76b9b7582e1b5ca830902dbc3a633baab9edde29c91602fa8cf34cbb48"} Oct 30 16:06:06.260630 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:06.260542 4401 generic.go:356] "Generic (PLEG): container finished" podID="e2cd27cc-3d42-4560-820e-015a5acac66b" containerID="6908cd9180ee5401d1974e8c25bc198cc5213f589df3f6d59b770e84a76fc12f" exitCode=0 Oct 30 16:06:06.261050 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:06.260628 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" event={"ID":"e2cd27cc-3d42-4560-820e-015a5acac66b","Type":"ContainerDied","Data":"6908cd9180ee5401d1974e8c25bc198cc5213f589df3f6d59b770e84a76fc12f"} Oct 30 16:06:07.425456 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.425430 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:07.571905 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.571786 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2cd27cc-3d42-4560-820e-015a5acac66b-kube-api-access-26dn7\" (UniqueName: \"kubernetes.io/projected/e2cd27cc-3d42-4560-820e-015a5acac66b-kube-api-access-26dn7\") pod \"e2cd27cc-3d42-4560-820e-015a5acac66b\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " Oct 30 16:06:07.571905 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.571884 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-openshift-ca\") pod \"e2cd27cc-3d42-4560-820e-015a5acac66b\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " Oct 30 16:06:07.572157 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.571975 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-xks-ca\") pod \"e2cd27cc-3d42-4560-820e-015a5acac66b\" (UID: \"e2cd27cc-3d42-4560-820e-015a5acac66b\") " Oct 30 16:06:07.572300 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.572266 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-openshift-ca" pod "e2cd27cc-3d42-4560-820e-015a5acac66b" (UID: "e2cd27cc-3d42-4560-820e-015a5acac66b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:06:07.574139 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.574109 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-xks-ca" pod "e2cd27cc-3d42-4560-820e-015a5acac66b" (UID: "e2cd27cc-3d42-4560-820e-015a5acac66b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:06:07.574265 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.574214 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2cd27cc-3d42-4560-820e-015a5acac66b-kube-api-access-26dn7" pod "e2cd27cc-3d42-4560-820e-015a5acac66b" (UID: "e2cd27cc-3d42-4560-820e-015a5acac66b"). InnerVolumeSpecName "kube-api-access-26dn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:06:07.673052 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.673020 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:07.673052 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.673048 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26dn7\" (UniqueName: \"kubernetes.io/projected/e2cd27cc-3d42-4560-820e-015a5acac66b-kube-api-access-26dn7\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:07.673322 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:07.673070 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2cd27cc-3d42-4560-820e-015a5acac66b-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:08.276057 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:08.276022 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" Oct 30 16:06:08.276239 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:08.276058 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb" event={"ID":"e2cd27cc-3d42-4560-820e-015a5acac66b","Type":"ContainerDied","Data":"b0d19e76b9b7582e1b5ca830902dbc3a633baab9edde29c91602fa8cf34cbb48"} Oct 30 16:06:08.276239 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:08.276139 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0d19e76b9b7582e1b5ca830902dbc3a633baab9edde29c91602fa8cf34cbb48" Oct 30 16:06:13.105643 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:13.105602 4401 scope.go:122] "RemoveContainer" containerID="d9af3519e915fbba0967ea4e9cbf106297cdd0dae2c04e8dd1682ca375ef7f1a" Oct 30 16:06:13.115365 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:13.115343 4401 scope.go:122] "RemoveContainer" containerID="e2ab850d7fce3a7f7f85b5a21328d3559328d49e0a0e7f2b1b9dfe972635af5c" Oct 30 16:06:13.126259 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:13.126238 4401 scope.go:122] "RemoveContainer" containerID="6bbad3f9ac183960acc1240af9659fc68ae1f86b267ef54590db85d7dea54358" Oct 30 16:06:15.834660 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:15.834624 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:06:15.973192 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:15.973160 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws"] Oct 30 16:06:15.973656 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:15.973643 4401 cpu_manager.go:414] "containerMap: removing container" podUID="e2cd27cc-3d42-4560-820e-015a5acac66b" containerName="revoke-keys" Oct 30 16:06:15.973700 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:15.973657 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2cd27cc-3d42-4560-820e-015a5acac66b" containerName="revoke-keys" Oct 30 16:06:15.973751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:15.973742 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2cd27cc-3d42-4560-820e-015a5acac66b" containerName="revoke-keys" Oct 30 16:06:15.978448 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:15.978422 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:15.996856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:15.996807 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws"] Oct 30 16:06:16.006282 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.006245 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:06:16.006428 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.006321 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:06:16.006428 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.006351 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:06:16.009174 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.009154 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:06:16.067857 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.067809 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.067857 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.067863 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdnl\" (UniqueName: \"kubernetes.io/projected/1d990338-8ca4-42d5-90b7-ec9d11566d53-kube-api-access-pzdnl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.068118 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.068032 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.168493 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.168457 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.168661 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.168565 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.168661 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.168599 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdnl\" (UniqueName: \"kubernetes.io/projected/1d990338-8ca4-42d5-90b7-ec9d11566d53-kube-api-access-pzdnl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.169266 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.169240 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.170947 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.170928 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.211760 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.211723 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdnl\" (UniqueName: \"kubernetes.io/projected/1d990338-8ca4-42d5-90b7-ec9d11566d53-kube-api-access-pzdnl\") pod \"maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.291888 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.291860 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:16.596655 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:16.596626 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws"] Oct 30 16:06:16.597552 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:06:16.597525 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d990338_8ca4_42d5_90b7_ec9d11566d53.slice/crio-a3ba0521730fc76c15d06b3c1a8c162d322fd2f33d4a10b2f7416f6ef5007de5 WatchSource:0}: Error finding container a3ba0521730fc76c15d06b3c1a8c162d322fd2f33d4a10b2f7416f6ef5007de5: Status 404 returned error can't find the container with id a3ba0521730fc76c15d06b3c1a8c162d322fd2f33d4a10b2f7416f6ef5007de5 Oct 30 16:06:17.332961 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:17.332867 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" event={"ID":"1d990338-8ca4-42d5-90b7-ec9d11566d53","Type":"ContainerStarted","Data":"f6549e681c054bffad2816e7607a4f332445183eee3c43109a84d58b4d456d86"} Oct 30 16:06:17.332961 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:17.332937 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" event={"ID":"1d990338-8ca4-42d5-90b7-ec9d11566d53","Type":"ContainerStarted","Data":"a3ba0521730fc76c15d06b3c1a8c162d322fd2f33d4a10b2f7416f6ef5007de5"} Oct 30 16:06:17.386979 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:17.386917 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" podStartSLOduration=2.386898746 podStartE2EDuration="2.386898746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:06:17.384743798 +0000 UTC m=+1686.084016955" watchObservedRunningTime="2026-10-30 16:06:17.386898746 +0000 UTC m=+1686.086171904" Oct 30 16:06:18.341660 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:18.341624 4401 generic.go:356] "Generic (PLEG): container finished" podID="1d990338-8ca4-42d5-90b7-ec9d11566d53" containerID="f6549e681c054bffad2816e7607a4f332445183eee3c43109a84d58b4d456d86" exitCode=0 Oct 30 16:06:18.342099 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:18.341716 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" event={"ID":"1d990338-8ca4-42d5-90b7-ec9d11566d53","Type":"ContainerDied","Data":"f6549e681c054bffad2816e7607a4f332445183eee3c43109a84d58b4d456d86"} Oct 30 16:06:19.481592 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.481567 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:19.509534 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.509506 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-xks-ca\") pod \"1d990338-8ca4-42d5-90b7-ec9d11566d53\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " Oct 30 16:06:19.509659 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.509552 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d990338-8ca4-42d5-90b7-ec9d11566d53-kube-api-access-pzdnl\" (UniqueName: \"kubernetes.io/projected/1d990338-8ca4-42d5-90b7-ec9d11566d53-kube-api-access-pzdnl\") pod \"1d990338-8ca4-42d5-90b7-ec9d11566d53\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " Oct 30 16:06:19.509731 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.509660 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-openshift-ca\") pod \"1d990338-8ca4-42d5-90b7-ec9d11566d53\" (UID: \"1d990338-8ca4-42d5-90b7-ec9d11566d53\") " Oct 30 16:06:19.510012 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.509986 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-openshift-ca" pod "1d990338-8ca4-42d5-90b7-ec9d11566d53" (UID: "1d990338-8ca4-42d5-90b7-ec9d11566d53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:06:19.511603 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.511574 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d990338-8ca4-42d5-90b7-ec9d11566d53-kube-api-access-pzdnl" pod "1d990338-8ca4-42d5-90b7-ec9d11566d53" (UID: "1d990338-8ca4-42d5-90b7-ec9d11566d53"). InnerVolumeSpecName "kube-api-access-pzdnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:06:19.511705 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.511653 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-xks-ca" pod "1d990338-8ca4-42d5-90b7-ec9d11566d53" (UID: "1d990338-8ca4-42d5-90b7-ec9d11566d53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:06:19.611162 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.611089 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:19.611162 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.611115 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzdnl\" (UniqueName: \"kubernetes.io/projected/1d990338-8ca4-42d5-90b7-ec9d11566d53-kube-api-access-pzdnl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:19.611162 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:19.611126 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d990338-8ca4-42d5-90b7-ec9d11566d53-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:20.356415 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:20.356383 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" Oct 30 16:06:20.356626 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:20.356418 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws" event={"ID":"1d990338-8ca4-42d5-90b7-ec9d11566d53","Type":"ContainerDied","Data":"a3ba0521730fc76c15d06b3c1a8c162d322fd2f33d4a10b2f7416f6ef5007de5"} Oct 30 16:06:20.356626 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:20.356464 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ba0521730fc76c15d06b3c1a8c162d322fd2f33d4a10b2f7416f6ef5007de5" Oct 30 16:06:20.740906 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:20.740813 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt"] Oct 30 16:06:21.965876 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:06:21.965767 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:06:26.459968 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:26.459938 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp"] Oct 30 16:06:26.501757 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:26.501725 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr"] Oct 30 16:06:30.382576 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:30.382543 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws"] Oct 30 16:06:30.395054 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:30.395027 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-6a2c74-9w9ws"] Oct 30 16:06:31.430926 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:31.430887 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d990338-8ca4-42d5-90b7-ec9d11566d53" path="/var/lib/kubelet/pods/1d990338-8ca4-42d5-90b7-ec9d11566d53/volumes" Oct 30 16:06:31.460485 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:31.460451 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" podUID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" containerName="payload-processing" containerID="cri-o://3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230" gracePeriod=25 Oct 30 16:06:31.502624 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:31.502595 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" podUID="916becdc-4e36-4eb5-b87d-6e1146d69acd" containerName="payload-pre-processing" containerID="cri-o://67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75" gracePeriod=25 Oct 30 16:06:32.038868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:32.037852 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:06:34.033060 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:34.033025 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm"] Oct 30 16:06:34.049363 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:34.049334 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-7fvzm"] Oct 30 16:06:35.427154 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:35.427116 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0106a30e-8004-451b-8c4c-70f31b5da50b" path="/var/lib/kubelet/pods/0106a30e-8004-451b-8c4c-70f31b5da50b/volumes" Oct 30 16:06:35.761860 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:35.761747 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" podUID="1a988a3f-15a6-4334-be34-ff54a7327010" containerName="main" containerID="cri-o://4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea" gracePeriod=45 Oct 30 16:06:36.029690 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.029663 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:06:36.191542 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191503 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-tmp-dir\") pod \"1a988a3f-15a6-4334-be34-ff54a7327010\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " Oct 30 16:06:36.191683 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191552 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a988a3f-15a6-4334-be34-ff54a7327010-kube-api-access-7zkhf\" (UniqueName: \"kubernetes.io/projected/1a988a3f-15a6-4334-be34-ff54a7327010-kube-api-access-7zkhf\") pod \"1a988a3f-15a6-4334-be34-ff54a7327010\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " Oct 30 16:06:36.191683 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191668 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-home\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-home\") pod \"1a988a3f-15a6-4334-be34-ff54a7327010\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " Oct 30 16:06:36.191792 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191729 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-dshm\") pod \"1a988a3f-15a6-4334-be34-ff54a7327010\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " Oct 30 16:06:36.191792 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191761 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a988a3f-15a6-4334-be34-ff54a7327010-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a988a3f-15a6-4334-be34-ff54a7327010-tls-certs\") pod \"1a988a3f-15a6-4334-be34-ff54a7327010\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " Oct 30 16:06:36.191792 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191766 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-tmp-dir" pod "1a988a3f-15a6-4334-be34-ff54a7327010" (UID: "1a988a3f-15a6-4334-be34-ff54a7327010"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:06:36.191995 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191872 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-model-cache\") pod \"1a988a3f-15a6-4334-be34-ff54a7327010\" (UID: \"1a988a3f-15a6-4334-be34-ff54a7327010\") " Oct 30 16:06:36.191995 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.191923 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-home" pod "1a988a3f-15a6-4334-be34-ff54a7327010" (UID: "1a988a3f-15a6-4334-be34-ff54a7327010"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:06:36.192140 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.192119 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-model-cache" pod "1a988a3f-15a6-4334-be34-ff54a7327010" (UID: "1a988a3f-15a6-4334-be34-ff54a7327010"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:06:36.192282 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.192264 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:36.192337 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.192288 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:36.192337 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.192301 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:36.193894 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.193866 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-dshm" pod "1a988a3f-15a6-4334-be34-ff54a7327010" (UID: "1a988a3f-15a6-4334-be34-ff54a7327010"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:06:36.193894 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.193879 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a988a3f-15a6-4334-be34-ff54a7327010-tls-certs" pod "1a988a3f-15a6-4334-be34-ff54a7327010" (UID: "1a988a3f-15a6-4334-be34-ff54a7327010"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:06:36.194037 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.193903 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a988a3f-15a6-4334-be34-ff54a7327010-kube-api-access-7zkhf" pod "1a988a3f-15a6-4334-be34-ff54a7327010" (UID: "1a988a3f-15a6-4334-be34-ff54a7327010"). InnerVolumeSpecName "kube-api-access-7zkhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:06:36.293247 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.293183 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a988a3f-15a6-4334-be34-ff54a7327010-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:36.293247 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.293205 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a988a3f-15a6-4334-be34-ff54a7327010-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:36.293247 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.293218 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zkhf\" (UniqueName: \"kubernetes.io/projected/1a988a3f-15a6-4334-be34-ff54a7327010-kube-api-access-7zkhf\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:36.462115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.462083 4401 generic.go:356] "Generic (PLEG): container finished" podID="1a988a3f-15a6-4334-be34-ff54a7327010" containerID="4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea" exitCode=0 Oct 30 16:06:36.462524 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.462130 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" Oct 30 16:06:36.462524 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.462162 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" event={"ID":"1a988a3f-15a6-4334-be34-ff54a7327010","Type":"ContainerDied","Data":"4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea"} Oct 30 16:06:36.462524 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.462202 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt" event={"ID":"1a988a3f-15a6-4334-be34-ff54a7327010","Type":"ContainerDied","Data":"022eaed9cc8729a27ee43c92c67bb7d7b62293f5075de5981de8110b9f9c9cd9"} Oct 30 16:06:36.462524 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.462221 4401 scope.go:122] "RemoveContainer" containerID="4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea" Oct 30 16:06:36.471884 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.471869 4401 scope.go:122] "RemoveContainer" containerID="4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea" Oct 30 16:06:36.472172 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:06:36.472154 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea\": container with ID starting with 4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea not found: ID does not exist" containerID="4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea" Oct 30 16:06:36.472223 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.472180 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea"} err="failed to get container status \"4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea\": rpc error: code = NotFound desc = could not find container \"4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea\": container with ID starting with 4b8b518140eb3f6ddb1501e4762f457aa9d2fd3f2162e58bf0f0e71ebf065fea not found: ID does not exist" Oct 30 16:06:36.523942 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.523917 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt"] Oct 30 16:06:36.537531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:36.537502 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-6a2c74/simulator-single-main-6a2c74-kserve-78d6856799-sgmmt"] Oct 30 16:06:37.015889 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:37.015854 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" podUID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:06:37.427267 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:37.427229 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a988a3f-15a6-4334-be34-ff54a7327010" path="/var/lib/kubelet/pods/1a988a3f-15a6-4334-be34-ff54a7327010/volumes" Oct 30 16:06:38.021429 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:38.021393 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" podUID="916becdc-4e36-4eb5-b87d-6e1146d69acd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:06:47.015914 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:47.015874 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" podUID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:06:48.022224 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:48.022185 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" podUID="916becdc-4e36-4eb5-b87d-6e1146d69acd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:06:51.418236 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.418202 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:06:51.780147 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.780121 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:06:51.796994 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.796968 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:06:51.859236 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.859201 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916becdc-4e36-4eb5-b87d-6e1146d69acd-kube-api-access-j5c4t\" (UniqueName: \"kubernetes.io/projected/916becdc-4e36-4eb5-b87d-6e1146d69acd-kube-api-access-j5c4t\") pod \"916becdc-4e36-4eb5-b87d-6e1146d69acd\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " Oct 30 16:06:51.859401 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.859316 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4e7c9c-fca4-4909-8bdc-1989807fd335-kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/4c4e7c9c-fca4-4909-8bdc-1989807fd335-kube-api-access-mtf2j\") pod \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " Oct 30 16:06:51.859463 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.859423 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4e7c9c-fca4-4909-8bdc-1989807fd335-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4e7c9c-fca4-4909-8bdc-1989807fd335-plugins-config-volume\") pod \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\" (UID: \"4c4e7c9c-fca4-4909-8bdc-1989807fd335\") " Oct 30 16:06:51.859514 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.859460 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916becdc-4e36-4eb5-b87d-6e1146d69acd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916becdc-4e36-4eb5-b87d-6e1146d69acd-plugins-config-volume\") pod \"916becdc-4e36-4eb5-b87d-6e1146d69acd\" (UID: \"916becdc-4e36-4eb5-b87d-6e1146d69acd\") " Oct 30 16:06:51.860614 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.860574 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916becdc-4e36-4eb5-b87d-6e1146d69acd-plugins-config-volume" pod "916becdc-4e36-4eb5-b87d-6e1146d69acd" (UID: "916becdc-4e36-4eb5-b87d-6e1146d69acd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:06:51.860814 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.860786 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4e7c9c-fca4-4909-8bdc-1989807fd335-plugins-config-volume" pod "4c4e7c9c-fca4-4909-8bdc-1989807fd335" (UID: "4c4e7c9c-fca4-4909-8bdc-1989807fd335"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:06:51.862771 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.862739 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4e7c9c-fca4-4909-8bdc-1989807fd335-kube-api-access-mtf2j" pod "4c4e7c9c-fca4-4909-8bdc-1989807fd335" (UID: "4c4e7c9c-fca4-4909-8bdc-1989807fd335"). InnerVolumeSpecName "kube-api-access-mtf2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:06:51.871544 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.871510 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916becdc-4e36-4eb5-b87d-6e1146d69acd-kube-api-access-j5c4t" pod "916becdc-4e36-4eb5-b87d-6e1146d69acd" (UID: "916becdc-4e36-4eb5-b87d-6e1146d69acd"). InnerVolumeSpecName "kube-api-access-j5c4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:06:51.960869 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.960811 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c4e7c9c-fca4-4909-8bdc-1989807fd335-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:51.961070 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.960933 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/916becdc-4e36-4eb5-b87d-6e1146d69acd-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:51.961070 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.960950 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5c4t\" (UniqueName: \"kubernetes.io/projected/916becdc-4e36-4eb5-b87d-6e1146d69acd-kube-api-access-j5c4t\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:51.961070 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:51.960964 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtf2j\" (UniqueName: \"kubernetes.io/projected/4c4e7c9c-fca4-4909-8bdc-1989807fd335-kube-api-access-mtf2j\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:06:51.979734 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:06:51.979643 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:06:52.620849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.618985 4401 generic.go:356] "Generic (PLEG): container finished" podID="916becdc-4e36-4eb5-b87d-6e1146d69acd" containerID="67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75" exitCode=0 Oct 30 16:06:52.620849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.619133 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" event={"ID":"916becdc-4e36-4eb5-b87d-6e1146d69acd","Type":"ContainerDied","Data":"67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75"} Oct 30 16:06:52.620849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.619208 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" event={"ID":"916becdc-4e36-4eb5-b87d-6e1146d69acd","Type":"ContainerDied","Data":"88bf1957e895fc32d697b6760dfd6e3d8c6ca3c893aedd383f5bb93a1c289d7f"} Oct 30 16:06:52.620849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.619260 4401 scope.go:122] "RemoveContainer" containerID="67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75" Oct 30 16:06:52.620849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.619508 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr" Oct 30 16:06:52.627619 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.627581 4401 generic.go:356] "Generic (PLEG): container finished" podID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" containerID="3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230" exitCode=0 Oct 30 16:06:52.628042 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.628022 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" Oct 30 16:06:52.637846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.628025 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" event={"ID":"4c4e7c9c-fca4-4909-8bdc-1989807fd335","Type":"ContainerDied","Data":"3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230"} Oct 30 16:06:52.637846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.636493 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp" event={"ID":"4c4e7c9c-fca4-4909-8bdc-1989807fd335","Type":"ContainerDied","Data":"7791aab6e0fb55c58f92085a1b482a1dd0eb453cca3fea08d7afaa5f8435f0ac"} Oct 30 16:06:52.716024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.715994 4401 scope.go:122] "RemoveContainer" containerID="67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75" Oct 30 16:06:52.716911 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:06:52.716864 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75\": container with ID starting with 67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75 not found: ID does not exist" containerID="67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75" Oct 30 16:06:52.717103 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.717073 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75"} err="failed to get container status \"67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75\": rpc error: code = NotFound desc = could not find container \"67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75\": container with ID starting with 67ed24e8875db678d9a7d91f03308add26767fc12f660b01dcc038f49d041e75 not found: ID does not exist" Oct 30 16:06:52.717229 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.717207 4401 scope.go:122] "RemoveContainer" containerID="3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230" Oct 30 16:06:52.791758 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.791625 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr"] Oct 30 16:06:52.823378 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.822633 4401 scope.go:122] "RemoveContainer" containerID="3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230" Oct 30 16:06:52.823542 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:06:52.823515 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230\": container with ID starting with 3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230 not found: ID does not exist" containerID="3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230" Oct 30 16:06:52.823618 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.823558 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230"} err="failed to get container status \"3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230\": rpc error: code = NotFound desc = could not find container \"3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230\": container with ID starting with 3e66fb1bb73e4eac157fcc26800baee4b85ab70fa39bb65581644134b1754230 not found: ID does not exist" Oct 30 16:06:52.831870 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.830869 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-6a2c74-64768cd5xr"] Oct 30 16:06:52.864960 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.864883 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp"] Oct 30 16:06:52.867603 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:52.867577 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6a2c74-7d94b64bc874lp"] Oct 30 16:06:53.436888 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:53.435896 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" path="/var/lib/kubelet/pods/4c4e7c9c-fca4-4909-8bdc-1989807fd335/volumes" Oct 30 16:06:53.436888 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:53.436482 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916becdc-4e36-4eb5-b87d-6e1146d69acd" path="/var/lib/kubelet/pods/916becdc-4e36-4eb5-b87d-6e1146d69acd/volumes" Oct 30 16:06:54.031034 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:54.031003 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5"] Oct 30 16:06:54.044851 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:54.044804 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-vw9s5"] Oct 30 16:06:55.427284 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:55.427252 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="050e229e-af32-4e27-aba9-b143cb16c679" path="/var/lib/kubelet/pods/050e229e-af32-4e27-aba9-b143cb16c679/volumes" Oct 30 16:06:57.202979 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:06:57.202940 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:07:06.031587 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:06.031547 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25"] Oct 30 16:07:06.045639 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:06.045609 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-bfw25"] Oct 30 16:07:07.427284 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:07.427249 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8423968c-a0fb-45b6-9c08-6d72d6c8bcf3" path="/var/lib/kubelet/pods/8423968c-a0fb-45b6-9c08-6d72d6c8bcf3/volumes" Oct 30 16:07:13.156023 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:13.155990 4401 scope.go:122] "RemoveContainer" containerID="7f4cab3a516d9d4c24d9a7a4fc35d03fe478d7f451dd803af9fd88da8cbde288" Oct 30 16:07:21.964999 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:07:21.964955 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:07:33.029761 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:33.029722 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5"] Oct 30 16:07:33.043540 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:33.043511 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-w4md5"] Oct 30 16:07:33.426645 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:33.426610 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20689837-6d7a-4b07-9d21-039f11a0f146" path="/var/lib/kubelet/pods/20689837-6d7a-4b07-9d21-039f11a0f146/volumes" Oct 30 16:07:44.037178 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:44.037145 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd"] Oct 30 16:07:44.051389 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:44.051356 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vltjd"] Oct 30 16:07:45.434264 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:07:45.434218 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9322250-35e6-428f-bcdf-67a08f4af1d3" path="/var/lib/kubelet/pods/e9322250-35e6-428f-bcdf-67a08f4af1d3/volumes" Oct 30 16:07:51.965325 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:07:51.965280 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:08:06.031495 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:06.031453 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst"] Oct 30 16:08:06.044205 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:06.044171 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-lbjst"] Oct 30 16:08:07.427124 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:07.427088 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba205c44-4823-4b48-b193-61b74f738f04" path="/var/lib/kubelet/pods/ba205c44-4823-4b48-b193-61b74f738f04/volumes" Oct 30 16:08:13.194602 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:13.194564 4401 scope.go:122] "RemoveContainer" containerID="34ff8e3c9c38041c29aaa5029085afc34d55cdc067c826376c2dc156e25934a8" Oct 30 16:08:13.203856 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:13.203815 4401 scope.go:122] "RemoveContainer" containerID="162b751aaa835561f864cfb40be6409d4d46bf872b7f14acf1e165a76a22f216" Oct 30 16:08:13.212739 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:13.212715 4401 scope.go:122] "RemoveContainer" containerID="f05ecf3d651caef7b07386b71d0829f241aad062735787f1ee622c2948f2ebd7" Oct 30 16:08:21.965883 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:08:21.965846 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:08:24.719014 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:24.718980 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:08:28.033082 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.033047 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm"] Oct 30 16:08:28.046623 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.046588 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zkkzm"] Oct 30 16:08:28.888901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.888870 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm"] Oct 30 16:08:28.889359 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889344 4401 cpu_manager.go:414] "containerMap: removing container" podUID="916becdc-4e36-4eb5-b87d-6e1146d69acd" containerName="payload-pre-processing" Oct 30 16:08:28.889411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889359 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916becdc-4e36-4eb5-b87d-6e1146d69acd" containerName="payload-pre-processing" Oct 30 16:08:28.889411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889373 4401 cpu_manager.go:414] "containerMap: removing container" podUID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" containerName="payload-processing" Oct 30 16:08:28.889411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889378 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" containerName="payload-processing" Oct 30 16:08:28.889411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889406 4401 cpu_manager.go:414] "containerMap: removing container" podUID="1d990338-8ca4-42d5-90b7-ec9d11566d53" containerName="revoke-keys" Oct 30 16:08:28.889411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889411 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d990338-8ca4-42d5-90b7-ec9d11566d53" containerName="revoke-keys" Oct 30 16:08:28.889599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889426 4401 cpu_manager.go:414] "containerMap: removing container" podUID="1a988a3f-15a6-4334-be34-ff54a7327010" containerName="main" Oct 30 16:08:28.889599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889431 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a988a3f-15a6-4334-be34-ff54a7327010" containerName="main" Oct 30 16:08:28.889599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889500 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a988a3f-15a6-4334-be34-ff54a7327010" containerName="main" Oct 30 16:08:28.889599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889512 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="916becdc-4e36-4eb5-b87d-6e1146d69acd" containerName="payload-pre-processing" Oct 30 16:08:28.889599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889519 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4e7c9c-fca4-4909-8bdc-1989807fd335" containerName="payload-processing" Oct 30 16:08:28.889599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.889528 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d990338-8ca4-42d5-90b7-ec9d11566d53" containerName="revoke-keys" Oct 30 16:08:28.894292 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.894270 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:28.906657 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.906632 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:28.906791 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.906752 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt5xh\" (UniqueName: \"kubernetes.io/projected/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-kube-api-access-wt5xh\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:28.906926 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.906796 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:28.911806 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.911784 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm"] Oct 30 16:08:28.922565 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.922540 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:08:28.924736 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.924718 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:08:28.924835 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.924765 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:08:28.924835 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:28.924799 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:08:29.007806 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.007774 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt5xh\" (UniqueName: \"kubernetes.io/projected/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-kube-api-access-wt5xh\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:29.007806 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.007808 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:29.008046 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.007930 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:29.008632 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.008607 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:29.010404 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.010381 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:29.044261 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.044235 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt5xh\" (UniqueName: \"kubernetes.io/projected/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-kube-api-access-wt5xh\") pod \"maas-api-revoke-sub-d576e68b-qwqzm\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:29.208072 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.207992 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:29.427917 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.427884 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ac591c-94da-464c-8bf1-a895223e78c2" path="/var/lib/kubelet/pods/39ac591c-94da-464c-8bf1-a895223e78c2/volumes" Oct 30 16:08:29.579577 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:29.579542 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm"] Oct 30 16:08:29.580933 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:08:29.580906 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab32563d_f5bf_4a7b_80c6_84d6d6c04e53.slice/crio-8ff7336c8c7ba930237983ae13768a2480b41774e2fe0686fae843d0da723935 WatchSource:0}: Error finding container 8ff7336c8c7ba930237983ae13768a2480b41774e2fe0686fae843d0da723935: Status 404 returned error can't find the container with id 8ff7336c8c7ba930237983ae13768a2480b41774e2fe0686fae843d0da723935 Oct 30 16:08:30.360947 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:30.360911 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" event={"ID":"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53","Type":"ContainerStarted","Data":"351c541d8a2de58467340763f6557cab04e76be8e1504806103fff27205226cb"} Oct 30 16:08:30.361326 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:30.360966 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" event={"ID":"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53","Type":"ContainerStarted","Data":"8ff7336c8c7ba930237983ae13768a2480b41774e2fe0686fae843d0da723935"} Oct 30 16:08:30.409087 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:30.409042 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" podStartSLOduration=2.409029167 podStartE2EDuration="2.409029167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:08:30.406965273 +0000 UTC m=+1819.106238431" watchObservedRunningTime="2026-10-30 16:08:30.409029167 +0000 UTC m=+1819.108302324" Oct 30 16:08:31.367986 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:31.367954 4401 generic.go:356] "Generic (PLEG): container finished" podID="ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" containerID="351c541d8a2de58467340763f6557cab04e76be8e1504806103fff27205226cb" exitCode=0 Oct 30 16:08:31.368411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:31.368047 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" event={"ID":"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53","Type":"ContainerDied","Data":"351c541d8a2de58467340763f6557cab04e76be8e1504806103fff27205226cb"} Oct 30 16:08:32.511115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.511092 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:32.538454 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.538431 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-kube-api-access-wt5xh\" (UniqueName: \"kubernetes.io/projected/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-kube-api-access-wt5xh\") pod \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " Oct 30 16:08:32.538585 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.538527 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-openshift-ca\") pod \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " Oct 30 16:08:32.538585 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.538561 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-xks-ca\") pod \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\" (UID: \"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53\") " Oct 30 16:08:32.538890 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.538862 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-openshift-ca" pod "ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" (UID: "ab32563d-f5bf-4a7b-80c6-84d6d6c04e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:08:32.540488 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.540465 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-xks-ca" pod "ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" (UID: "ab32563d-f5bf-4a7b-80c6-84d6d6c04e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:08:32.540605 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.540466 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-kube-api-access-wt5xh" pod "ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" (UID: "ab32563d-f5bf-4a7b-80c6-84d6d6c04e53"). InnerVolumeSpecName "kube-api-access-wt5xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:08:32.639520 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.639438 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:08:32.639520 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.639465 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:08:32.639520 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:32.639481 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt5xh\" (UniqueName: \"kubernetes.io/projected/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53-kube-api-access-wt5xh\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:08:33.387684 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:33.387653 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" Oct 30 16:08:33.387888 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:33.387689 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm" event={"ID":"ab32563d-f5bf-4a7b-80c6-84d6d6c04e53","Type":"ContainerDied","Data":"8ff7336c8c7ba930237983ae13768a2480b41774e2fe0686fae843d0da723935"} Oct 30 16:08:33.387888 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:33.387764 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff7336c8c7ba930237983ae13768a2480b41774e2fe0686fae843d0da723935" Oct 30 16:08:40.095704 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:40.095668 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:08:49.033907 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:49.033869 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb"] Oct 30 16:08:49.047815 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:49.047790 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mv7jb"] Oct 30 16:08:49.426697 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:08:49.426663 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcc3e36-c0f5-4012-a058-eee8f23eab7b" path="/var/lib/kubelet/pods/2bcc3e36-c0f5-4012-a058-eee8f23eab7b/volumes" Oct 30 16:08:51.965809 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:08:51.965772 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:09:03.300113 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.300037 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt"] Oct 30 16:09:03.300552 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.300535 4401 cpu_manager.go:414] "containerMap: removing container" podUID="ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" containerName="revoke-keys" Oct 30 16:09:03.300596 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.300554 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" containerName="revoke-keys" Oct 30 16:09:03.300650 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.300641 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" containerName="revoke-keys" Oct 30 16:09:03.304960 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.304939 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.324134 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.324109 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt"] Oct 30 16:09:03.335098 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.335072 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d44b7c21-dockercfg-7zjnd\"" Oct 30 16:09:03.335214 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.335160 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-df488b\"" Oct 30 16:09:03.368118 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.368091 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb"] Oct 30 16:09:03.371739 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.371714 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.389905 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.389869 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb"] Oct 30 16:09:03.405702 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.405676 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfl4k\" (UniqueName: \"kubernetes.io/projected/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-kube-api-access-bfl4k\") pod \"payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.405842 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.405735 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb79c17-7e58-49b8-a342-52a42af7ae97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.405898 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.405808 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2c7j\" (UniqueName: \"kubernetes.io/projected/6eb79c17-7e58-49b8-a342-52a42af7ae97-kube-api-access-c2c7j\") pod \"payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.405950 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.405932 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.506526 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.506494 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.506706 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.506577 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfl4k\" (UniqueName: \"kubernetes.io/projected/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-kube-api-access-bfl4k\") pod \"payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.506706 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.506629 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb79c17-7e58-49b8-a342-52a42af7ae97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.506706 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.506679 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2c7j\" (UniqueName: \"kubernetes.io/projected/6eb79c17-7e58-49b8-a342-52a42af7ae97-kube-api-access-c2c7j\") pod \"payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.507297 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.507270 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.507420 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.507394 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb79c17-7e58-49b8-a342-52a42af7ae97-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.531705 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.531674 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2c7j\" (UniqueName: \"kubernetes.io/projected/6eb79c17-7e58-49b8-a342-52a42af7ae97-kube-api-access-c2c7j\") pod \"payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.531880 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.531858 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfl4k\" (UniqueName: \"kubernetes.io/projected/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-kube-api-access-bfl4k\") pod \"payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.618633 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.618605 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:03.685306 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.685271 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:03.922501 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.922475 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt"] Oct 30 16:09:03.923383 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:09:03.923355 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8536cd6c_0fb0_470c_9a6d_d38ca7d5a35f.slice/crio-2da83ca162850441d7ba1d44315bb1d94f9738d6735c26b399838ac376f2b9f0 WatchSource:0}: Error finding container 2da83ca162850441d7ba1d44315bb1d94f9738d6735c26b399838ac376f2b9f0: Status 404 returned error can't find the container with id 2da83ca162850441d7ba1d44315bb1d94f9738d6735c26b399838ac376f2b9f0 Oct 30 16:09:03.998314 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:09:03.998290 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb79c17_7e58_49b8_a342_52a42af7ae97.slice/crio-1db15a148f380cf359f6afdc42a8a65d9414126fbef48309be9e64732b2460a9 WatchSource:0}: Error finding container 1db15a148f380cf359f6afdc42a8a65d9414126fbef48309be9e64732b2460a9: Status 404 returned error can't find the container with id 1db15a148f380cf359f6afdc42a8a65d9414126fbef48309be9e64732b2460a9 Oct 30 16:09:03.999195 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:03.999171 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb"] Oct 30 16:09:04.594227 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.594186 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" event={"ID":"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f","Type":"ContainerStarted","Data":"6abd912b1628e27bcda36500bdd393019bf2afaeb3bac7fd06594a4f3c4821ce"} Oct 30 16:09:04.594681 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.594249 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" event={"ID":"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f","Type":"ContainerStarted","Data":"2da83ca162850441d7ba1d44315bb1d94f9738d6735c26b399838ac376f2b9f0"} Oct 30 16:09:04.594681 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.594385 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:04.595784 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.595763 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" event={"ID":"6eb79c17-7e58-49b8-a342-52a42af7ae97","Type":"ContainerStarted","Data":"ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc"} Oct 30 16:09:04.595888 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.595799 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" event={"ID":"6eb79c17-7e58-49b8-a342-52a42af7ae97","Type":"ContainerStarted","Data":"1db15a148f380cf359f6afdc42a8a65d9414126fbef48309be9e64732b2460a9"} Oct 30 16:09:04.595888 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.595819 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:04.640196 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.640147 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" podStartSLOduration=1.6401345040000002 podStartE2EDuration="1.640134504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:09:04.637436495 +0000 UTC m=+1853.336709652" watchObservedRunningTime="2026-10-30 16:09:04.640134504 +0000 UTC m=+1853.339407662" Oct 30 16:09:04.684606 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:04.684550 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" podStartSLOduration=1.684536655 podStartE2EDuration="1.684536655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:09:04.683629958 +0000 UTC m=+1853.382903115" watchObservedRunningTime="2026-10-30 16:09:04.684536655 +0000 UTC m=+1853.383809809" Oct 30 16:09:05.030180 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:05.030147 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485"] Oct 30 16:09:05.032764 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:05.032742 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-2w485"] Oct 30 16:09:05.427441 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:05.427404 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7931d90-a646-44df-94e3-70efb20efad0" path="/var/lib/kubelet/pods/d7931d90-a646-44df-94e3-70efb20efad0/volumes" Oct 30 16:09:06.802957 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:06.802905 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:09:13.247644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:13.247604 4401 scope.go:122] "RemoveContainer" containerID="fec69a83107e894780db77e646c48373aacf17bf1dbc763e3a5a8c0bd6c73b22" Oct 30 16:09:13.259040 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:13.259021 4401 scope.go:122] "RemoveContainer" containerID="624ff63bda3e186dfd3e43e79f9d262cf4877b98cb314073dc1ced4bfdf7ab75" Oct 30 16:09:13.267174 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:13.267151 4401 scope.go:122] "RemoveContainer" containerID="89c20e57532f30636928cc86320d770d44b90c8593983171ace3aa012c41de3b" Oct 30 16:09:15.605352 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:15.605322 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:09:15.605850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:15.605385 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:09:19.607955 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.607918 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h"] Oct 30 16:09:19.613716 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.613688 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.632215 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.632192 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h"] Oct 30 16:09:19.644247 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.644220 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-df488b\"/\"openshift-service-ca.crt\"" Oct 30 16:09:19.644415 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.644400 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-df488b\"/\"default-dockercfg-vxtnd\"" Oct 30 16:09:19.645022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.644995 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-df488b\"/\"kube-root-ca.crt\"" Oct 30 16:09:19.647353 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.647333 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-df488b\"/\"simulator-single-main-df488b-kserve-self-signed-certs\"" Oct 30 16:09:19.669099 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.669070 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9hcj\" (UniqueName: \"kubernetes.io/projected/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-kube-api-access-p9hcj\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.669229 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.669126 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-home\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.669229 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.669162 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tls-certs\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.669229 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.669199 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-dshm\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.669381 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.669309 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-model-cache\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.669434 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.669397 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tmp-dir\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770250 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770216 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-model-cache\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770409 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770293 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tmp-dir\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770409 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770333 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9hcj\" (UniqueName: \"kubernetes.io/projected/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-kube-api-access-p9hcj\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770409 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770369 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-home\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770409 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770398 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tls-certs\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770626 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770427 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-dshm\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770745 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770709 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-model-cache\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770865 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770817 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tmp-dir\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.770941 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.770836 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-home\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.772749 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.772726 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-dshm\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.773047 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.773031 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tls-certs\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.795299 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.795277 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9hcj\" (UniqueName: \"kubernetes.io/projected/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-kube-api-access-p9hcj\") pod \"simulator-single-main-df488b-kserve-556b495c86-4bc4h\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:19.926481 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:19.926395 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:20.248849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:20.248793 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h"] Oct 30 16:09:20.248967 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:09:20.248795 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3eca5425_2f6b_43aa_9705_8e972a3b4c2b.slice/crio-8bc0aea41283a92b1177c06d229894d49e269a9ebd30b382d56f8f50de8e98f1 WatchSource:0}: Error finding container 8bc0aea41283a92b1177c06d229894d49e269a9ebd30b382d56f8f50de8e98f1: Status 404 returned error can't find the container with id 8bc0aea41283a92b1177c06d229894d49e269a9ebd30b382d56f8f50de8e98f1 Oct 30 16:09:20.691886 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:20.691852 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" event={"ID":"3eca5425-2f6b-43aa-9705-8e972a3b4c2b","Type":"ContainerStarted","Data":"d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2"} Oct 30 16:09:20.692280 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:20.691894 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" event={"ID":"3eca5425-2f6b-43aa-9705-8e972a3b4c2b","Type":"ContainerStarted","Data":"8bc0aea41283a92b1177c06d229894d49e269a9ebd30b382d56f8f50de8e98f1"} Oct 30 16:09:20.737730 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:20.737685 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" podStartSLOduration=1.737672983 podStartE2EDuration="1.737672983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:09:20.736039907 +0000 UTC m=+1869.435313063" watchObservedRunningTime="2026-10-30 16:09:20.737672983 +0000 UTC m=+1869.436946139" Oct 30 16:09:20.927076 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:20.927039 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:21.966375 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:09:21.966341 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:09:22.097041 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:22.097002 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:09:29.927603 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:29.927564 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:29.939988 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:29.939963 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:30.759571 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:30.759538 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:09:36.323133 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:36.323100 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:09:38.472098 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.472060 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p"] Oct 30 16:09:38.475735 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.475711 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.494476 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.494449 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p"] Oct 30 16:09:38.504382 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.504350 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:09:38.504382 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.504359 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:09:38.504533 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.504367 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:09:38.506198 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.506177 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:09:38.565407 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.565370 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.565573 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.565555 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtcvv\" (UniqueName: \"kubernetes.io/projected/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-kube-api-access-vtcvv\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.565639 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.565618 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.666558 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.666500 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtcvv\" (UniqueName: \"kubernetes.io/projected/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-kube-api-access-vtcvv\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.666798 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.666615 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.666798 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.666720 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.667636 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.667610 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.669260 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.669242 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.697034 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.697002 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtcvv\" (UniqueName: \"kubernetes.io/projected/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-kube-api-access-vtcvv\") pod \"maas-api-revoke-sub-3bd20fdf-q524p\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:38.794023 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:38.793942 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:39.098138 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:39.098099 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p"] Oct 30 16:09:39.099210 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:09:39.099166 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3dcbad_d699_4d0b_b7a8_8c509b10b5bc.slice/crio-655a003fa3aab319a5029f4ff7eac3d6e2d63e3fd231f645315636034516f54f WatchSource:0}: Error finding container 655a003fa3aab319a5029f4ff7eac3d6e2d63e3fd231f645315636034516f54f: Status 404 returned error can't find the container with id 655a003fa3aab319a5029f4ff7eac3d6e2d63e3fd231f645315636034516f54f Oct 30 16:09:39.807865 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:39.807768 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" event={"ID":"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc","Type":"ContainerStarted","Data":"ad82cb595d987ba080bf65cd6ed60fa1bfeb983abbe3e7af09bd5351c7c88478"} Oct 30 16:09:39.807865 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:39.807836 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" event={"ID":"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc","Type":"ContainerStarted","Data":"655a003fa3aab319a5029f4ff7eac3d6e2d63e3fd231f645315636034516f54f"} Oct 30 16:09:39.853227 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:39.853180 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" podStartSLOduration=1.8531613 podStartE2EDuration="1.8531613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:09:39.850882048 +0000 UTC m=+1888.550155218" watchObservedRunningTime="2026-10-30 16:09:39.8531613 +0000 UTC m=+1888.552434490" Oct 30 16:09:40.816302 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:40.816264 4401 generic.go:356] "Generic (PLEG): container finished" podID="ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" containerID="ad82cb595d987ba080bf65cd6ed60fa1bfeb983abbe3e7af09bd5351c7c88478" exitCode=0 Oct 30 16:09:40.816700 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:40.816296 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" event={"ID":"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc","Type":"ContainerDied","Data":"ad82cb595d987ba080bf65cd6ed60fa1bfeb983abbe3e7af09bd5351c7c88478"} Oct 30 16:09:41.965589 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:41.965565 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:42.107510 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.107480 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-kube-api-access-vtcvv\" (UniqueName: \"kubernetes.io/projected/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-kube-api-access-vtcvv\") pod \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " Oct 30 16:09:42.107669 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.107617 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-xks-ca\") pod \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " Oct 30 16:09:42.107708 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.107672 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-openshift-ca\") pod \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\" (UID: \"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc\") " Oct 30 16:09:42.108068 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.108042 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-openshift-ca" pod "ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" (UID: "ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:09:42.109628 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.109598 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-xks-ca" pod "ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" (UID: "ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:09:42.109739 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.109723 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-kube-api-access-vtcvv" pod "ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" (UID: "ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc"). InnerVolumeSpecName "kube-api-access-vtcvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:09:42.212208 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.210146 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:09:42.212208 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.210186 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:09:42.212208 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.210200 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtcvv\" (UniqueName: \"kubernetes.io/projected/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc-kube-api-access-vtcvv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:09:42.829738 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.829699 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" Oct 30 16:09:42.829738 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.829730 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p" event={"ID":"ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc","Type":"ContainerDied","Data":"655a003fa3aab319a5029f4ff7eac3d6e2d63e3fd231f645315636034516f54f"} Oct 30 16:09:42.829966 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:42.829777 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="655a003fa3aab319a5029f4ff7eac3d6e2d63e3fd231f645315636034516f54f" Oct 30 16:09:49.577816 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.577771 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl"] Oct 30 16:09:49.578458 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.578245 4401 cpu_manager.go:414] "containerMap: removing container" podUID="ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" containerName="revoke-keys" Oct 30 16:09:49.578458 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.578257 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" containerName="revoke-keys" Oct 30 16:09:49.578458 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.578361 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" containerName="revoke-keys" Oct 30 16:09:49.582124 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.582098 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.600150 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.600121 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl"] Oct 30 16:09:49.610264 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.610213 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:09:49.610264 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.610220 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:09:49.610456 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.610221 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:09:49.612427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.612409 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:09:49.682507 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.682477 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.682671 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.682542 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/eb59516d-e025-4a6f-8f46-a74ce990f749-kube-api-access-fkwxx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.682671 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.682564 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.784138 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.784101 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.784381 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.784192 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/eb59516d-e025-4a6f-8f46-a74ce990f749-kube-api-access-fkwxx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.784381 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.784214 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.784730 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.784709 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.786562 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.786539 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.809350 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.809317 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/eb59516d-e025-4a6f-8f46-a74ce990f749-kube-api-access-fkwxx\") pod \"maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:49.882635 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.882596 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:09:49.895430 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:49.895401 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:50.207772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:50.207745 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl"] Oct 30 16:09:50.209193 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:09:50.209156 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb59516d_e025_4a6f_8f46_a74ce990f749.slice/crio-f033bde08e66f33703fae14bb8812b2d7688f63b15449feab8bf5572cb707100 WatchSource:0}: Error finding container f033bde08e66f33703fae14bb8812b2d7688f63b15449feab8bf5572cb707100: Status 404 returned error can't find the container with id f033bde08e66f33703fae14bb8812b2d7688f63b15449feab8bf5572cb707100 Oct 30 16:09:50.886103 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:50.886062 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" event={"ID":"eb59516d-e025-4a6f-8f46-a74ce990f749","Type":"ContainerStarted","Data":"d55e6848276cb3c8f48f7fdf654516191e26ab19e0c2dbef89fd225b81067195"} Oct 30 16:09:50.886623 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:50.886119 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" event={"ID":"eb59516d-e025-4a6f-8f46-a74ce990f749","Type":"ContainerStarted","Data":"f033bde08e66f33703fae14bb8812b2d7688f63b15449feab8bf5572cb707100"} Oct 30 16:09:50.933066 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:50.933020 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" podStartSLOduration=1.93300744 podStartE2EDuration="1.93300744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:09:50.93180985 +0000 UTC m=+1899.631083007" watchObservedRunningTime="2026-10-30 16:09:50.93300744 +0000 UTC m=+1899.632280596" Oct 30 16:09:51.893264 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:51.893224 4401 generic.go:356] "Generic (PLEG): container finished" podID="eb59516d-e025-4a6f-8f46-a74ce990f749" containerID="d55e6848276cb3c8f48f7fdf654516191e26ab19e0c2dbef89fd225b81067195" exitCode=0 Oct 30 16:09:51.893723 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:51.893315 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" event={"ID":"eb59516d-e025-4a6f-8f46-a74ce990f749","Type":"ContainerDied","Data":"d55e6848276cb3c8f48f7fdf654516191e26ab19e0c2dbef89fd225b81067195"} Oct 30 16:09:51.968857 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:09:51.968805 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:09:53.040140 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.040113 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:53.126971 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.126939 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-openshift-ca\") pod \"eb59516d-e025-4a6f-8f46-a74ce990f749\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " Oct 30 16:09:53.127131 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.127041 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-xks-ca\") pod \"eb59516d-e025-4a6f-8f46-a74ce990f749\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " Oct 30 16:09:53.127131 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.127072 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb59516d-e025-4a6f-8f46-a74ce990f749-kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/eb59516d-e025-4a6f-8f46-a74ce990f749-kube-api-access-fkwxx\") pod \"eb59516d-e025-4a6f-8f46-a74ce990f749\" (UID: \"eb59516d-e025-4a6f-8f46-a74ce990f749\") " Oct 30 16:09:53.127351 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.127328 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-openshift-ca" pod "eb59516d-e025-4a6f-8f46-a74ce990f749" (UID: "eb59516d-e025-4a6f-8f46-a74ce990f749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:09:53.127446 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.127432 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:09:53.129186 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.129156 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-xks-ca" pod "eb59516d-e025-4a6f-8f46-a74ce990f749" (UID: "eb59516d-e025-4a6f-8f46-a74ce990f749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:09:53.129254 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.129159 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb59516d-e025-4a6f-8f46-a74ce990f749-kube-api-access-fkwxx" pod "eb59516d-e025-4a6f-8f46-a74ce990f749" (UID: "eb59516d-e025-4a6f-8f46-a74ce990f749"). InnerVolumeSpecName "kube-api-access-fkwxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:09:53.228166 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.228104 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb59516d-e025-4a6f-8f46-a74ce990f749-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:09:53.228166 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.228129 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkwxx\" (UniqueName: \"kubernetes.io/projected/eb59516d-e025-4a6f-8f46-a74ce990f749-kube-api-access-fkwxx\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:09:53.906346 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.906312 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" Oct 30 16:09:53.906531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.906350 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl" event={"ID":"eb59516d-e025-4a6f-8f46-a74ce990f749","Type":"ContainerDied","Data":"f033bde08e66f33703fae14bb8812b2d7688f63b15449feab8bf5572cb707100"} Oct 30 16:09:53.906531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:53.906407 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f033bde08e66f33703fae14bb8812b2d7688f63b15449feab8bf5572cb707100" Oct 30 16:09:54.351772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:09:54.351740 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h"] Oct 30 16:10:00.031994 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:00.031962 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt"] Oct 30 16:10:00.047101 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:00.047067 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb"] Oct 30 16:10:03.659838 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:03.659797 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl"] Oct 30 16:10:03.681174 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:03.681124 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-df488b-bkfnl"] Oct 30 16:10:05.032770 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:05.032732 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" podUID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" containerName="payload-processing" containerID="cri-o://6abd912b1628e27bcda36500bdd393019bf2afaeb3bac7fd06594a4f3c4821ce" gracePeriod=25 Oct 30 16:10:05.047928 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:05.047876 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" podUID="6eb79c17-7e58-49b8-a342-52a42af7ae97" containerName="payload-pre-processing" containerID="cri-o://ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc" gracePeriod=25 Oct 30 16:10:05.129956 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:05.129592 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:10:05.427125 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:05.427091 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb59516d-e025-4a6f-8f46-a74ce990f749" path="/var/lib/kubelet/pods/eb59516d-e025-4a6f-8f46-a74ce990f749/volumes" Oct 30 16:10:05.604459 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:05.604418 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" podUID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:10:05.604679 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:05.604434 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" podUID="6eb79c17-7e58-49b8-a342-52a42af7ae97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:10:09.372381 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.372316 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" podUID="3eca5425-2f6b-43aa-9705-8e972a3b4c2b" containerName="main" containerID="cri-o://d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2" gracePeriod=45 Oct 30 16:10:09.625514 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.625456 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:10:09.695952 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.695921 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tmp-dir\") pod \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " Oct 30 16:10:09.696154 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.695964 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-kube-api-access-p9hcj\" (UniqueName: \"kubernetes.io/projected/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-kube-api-access-p9hcj\") pod \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " Oct 30 16:10:09.696154 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696072 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tls-certs\") pod \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " Oct 30 16:10:09.696294 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696213 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-dshm\") pod \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " Oct 30 16:10:09.696294 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696233 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tmp-dir" pod "3eca5425-2f6b-43aa-9705-8e972a3b4c2b" (UID: "3eca5425-2f6b-43aa-9705-8e972a3b4c2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:10:09.696294 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696285 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-home\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-home\") pod \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " Oct 30 16:10:09.696464 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696367 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-model-cache\") pod \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\" (UID: \"3eca5425-2f6b-43aa-9705-8e972a3b4c2b\") " Oct 30 16:10:09.696842 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696783 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:09.696842 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696763 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-home" pod "3eca5425-2f6b-43aa-9705-8e972a3b4c2b" (UID: "3eca5425-2f6b-43aa-9705-8e972a3b4c2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:10:09.697038 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.696852 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-model-cache" pod "3eca5425-2f6b-43aa-9705-8e972a3b4c2b" (UID: "3eca5425-2f6b-43aa-9705-8e972a3b4c2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:10:09.699289 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.699251 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-dshm" pod "3eca5425-2f6b-43aa-9705-8e972a3b4c2b" (UID: "3eca5425-2f6b-43aa-9705-8e972a3b4c2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:10:09.699289 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.699262 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-kube-api-access-p9hcj" pod "3eca5425-2f6b-43aa-9705-8e972a3b4c2b" (UID: "3eca5425-2f6b-43aa-9705-8e972a3b4c2b"). InnerVolumeSpecName "kube-api-access-p9hcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:10:09.699543 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.699519 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tls-certs" pod "3eca5425-2f6b-43aa-9705-8e972a3b4c2b" (UID: "3eca5425-2f6b-43aa-9705-8e972a3b4c2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:10:09.798333 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.798297 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9hcj\" (UniqueName: \"kubernetes.io/projected/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-kube-api-access-p9hcj\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:09.798333 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.798327 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:09.798333 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.798338 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:09.798570 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.798346 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:09.798570 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:09.798354 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3eca5425-2f6b-43aa-9705-8e972a3b4c2b-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:10.007908 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.007794 4401 generic.go:356] "Generic (PLEG): container finished" podID="3eca5425-2f6b-43aa-9705-8e972a3b4c2b" containerID="d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2" exitCode=0 Oct 30 16:10:10.007908 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.007867 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" Oct 30 16:10:10.007908 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.007886 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" event={"ID":"3eca5425-2f6b-43aa-9705-8e972a3b4c2b","Type":"ContainerDied","Data":"d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2"} Oct 30 16:10:10.008160 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.007932 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h" event={"ID":"3eca5425-2f6b-43aa-9705-8e972a3b4c2b","Type":"ContainerDied","Data":"8bc0aea41283a92b1177c06d229894d49e269a9ebd30b382d56f8f50de8e98f1"} Oct 30 16:10:10.008160 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.007949 4401 scope.go:122] "RemoveContainer" containerID="d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2" Oct 30 16:10:10.018075 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.018049 4401 scope.go:122] "RemoveContainer" containerID="d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2" Oct 30 16:10:10.018353 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:10:10.018333 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2\": container with ID starting with d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2 not found: ID does not exist" containerID="d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2" Oct 30 16:10:10.018404 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.018362 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2"} err="failed to get container status \"d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2\": rpc error: code = NotFound desc = could not find container \"d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2\": container with ID starting with d7b163c00fb235a0e0260413a515e44bff91be9b7e9f82fbb9fce7b9c3ee71c2 not found: ID does not exist" Oct 30 16:10:10.069494 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.069459 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h"] Oct 30 16:10:10.072788 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:10.072764 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-df488b/simulator-single-main-df488b-kserve-556b495c86-4bc4h"] Oct 30 16:10:11.427553 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:11.427522 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eca5425-2f6b-43aa-9705-8e972a3b4c2b" path="/var/lib/kubelet/pods/3eca5425-2f6b-43aa-9705-8e972a3b4c2b/volumes" Oct 30 16:10:13.296644 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:13.296603 4401 scope.go:122] "RemoveContainer" containerID="e9d6770b3480f425edeb32c2c29e10d4f570ecf7c0123a596baa3c604fcadf17" Oct 30 16:10:13.305434 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:13.305400 4401 scope.go:122] "RemoveContainer" containerID="63c38758b4dd16d423f00d261a83f50edadc1d9525c0cdea1a6934dee1231122" Oct 30 16:10:13.314983 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:13.314957 4401 scope.go:122] "RemoveContainer" containerID="88004361ec4acd79fcb893eb50e31c52e8b041ad4a65a28f11029ec88ea1a874" Oct 30 16:10:15.239953 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.239918 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4"] Oct 30 16:10:15.240459 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.240375 4401 cpu_manager.go:414] "containerMap: removing container" podUID="eb59516d-e025-4a6f-8f46-a74ce990f749" containerName="revoke-keys" Oct 30 16:10:15.240459 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.240387 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb59516d-e025-4a6f-8f46-a74ce990f749" containerName="revoke-keys" Oct 30 16:10:15.240459 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.240398 4401 cpu_manager.go:414] "containerMap: removing container" podUID="3eca5425-2f6b-43aa-9705-8e972a3b4c2b" containerName="main" Oct 30 16:10:15.240459 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.240403 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eca5425-2f6b-43aa-9705-8e972a3b4c2b" containerName="main" Oct 30 16:10:15.240705 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.240476 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb59516d-e025-4a6f-8f46-a74ce990f749" containerName="revoke-keys" Oct 30 16:10:15.240705 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.240486 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eca5425-2f6b-43aa-9705-8e972a3b4c2b" containerName="main" Oct 30 16:10:15.245372 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.245349 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.263768 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.263746 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4"] Oct 30 16:10:15.273625 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.273608 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3c05ee64-kserve-self-signed-certs\"" Oct 30 16:10:15.353902 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.353871 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-tmp-dir\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.354051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.353923 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-home\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.354051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.354013 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c35e2df-7c75-41c7-bfd5-25081925852e-tls-certs\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.354051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.354035 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-dshm\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.354159 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.354084 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgl2d\" (UniqueName: \"kubernetes.io/projected/7c35e2df-7c75-41c7-bfd5-25081925852e-kube-api-access-zgl2d\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.354206 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.354186 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-model-cache\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.455850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.455591 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-model-cache\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.455850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.455720 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-tmp-dir\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.455850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.455777 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-home\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.455850 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.455851 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c35e2df-7c75-41c7-bfd5-25081925852e-tls-certs\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.456235 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.455876 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-dshm\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.456235 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.455947 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgl2d\" (UniqueName: \"kubernetes.io/projected/7c35e2df-7c75-41c7-bfd5-25081925852e-kube-api-access-zgl2d\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.456701 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.456672 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-model-cache\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.457219 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.457199 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-tmp-dir\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.457573 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.457554 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-home\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.461126 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.461105 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-dshm\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.461267 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.461180 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c35e2df-7c75-41c7-bfd5-25081925852e-tls-certs\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.481183 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.481146 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgl2d\" (UniqueName: \"kubernetes.io/projected/7c35e2df-7c75-41c7-bfd5-25081925852e-kube-api-access-zgl2d\") pod \"e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.558566 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.558505 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:15.604637 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.604604 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" podUID="6eb79c17-7e58-49b8-a342-52a42af7ae97" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:10:15.604884 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.604676 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" podUID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 16:10:15.865255 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:15.865221 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4"] Oct 30 16:10:15.866376 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:10:15.866350 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c35e2df_7c75_41c7_bfd5_25081925852e.slice/crio-d6380236c210e33d0b089a8555d66f996663dca35db092c6552d4c1469a6fece WatchSource:0}: Error finding container d6380236c210e33d0b089a8555d66f996663dca35db092c6552d4c1469a6fece: Status 404 returned error can't find the container with id d6380236c210e33d0b089a8555d66f996663dca35db092c6552d4c1469a6fece Oct 30 16:10:16.045673 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:16.045632 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" event={"ID":"7c35e2df-7c75-41c7-bfd5-25081925852e","Type":"ContainerStarted","Data":"8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96"} Oct 30 16:10:16.045885 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:16.045686 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" event={"ID":"7c35e2df-7c75-41c7-bfd5-25081925852e","Type":"ContainerStarted","Data":"d6380236c210e33d0b089a8555d66f996663dca35db092c6552d4c1469a6fece"} Oct 30 16:10:16.090199 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:16.090150 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" podStartSLOduration=1.090137944 podStartE2EDuration="1.090137944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:10:16.089700713 +0000 UTC m=+1924.788973870" watchObservedRunningTime="2026-10-30 16:10:16.090137944 +0000 UTC m=+1924.789411102" Oct 30 16:10:16.559271 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:16.559226 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:17.310443 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:17.310414 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:10:20.463088 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.463037 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:10:20.463759 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.463708 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:10:20.505077 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.504913 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:10:20.505077 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.505017 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:10:20.629141 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.629087 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:10:20.629479 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.629429 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:10:20.735024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.731940 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:10:20.735024 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:20.732034 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:10:21.965472 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:10:21.965435 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:10:25.161990 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.161957 4401 generic.go:356] "Generic (PLEG): container finished" podID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" containerID="6abd912b1628e27bcda36500bdd393019bf2afaeb3bac7fd06594a4f3c4821ce" exitCode=0 Oct 30 16:10:25.162332 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.162101 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" event={"ID":"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f","Type":"ContainerDied","Data":"6abd912b1628e27bcda36500bdd393019bf2afaeb3bac7fd06594a4f3c4821ce"} Oct 30 16:10:25.307098 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.307069 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:10:25.338076 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.338053 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:10:25.389540 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.389510 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-kube-api-access-bfl4k\" (UniqueName: \"kubernetes.io/projected/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-kube-api-access-bfl4k\") pod \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " Oct 30 16:10:25.389690 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.389561 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-plugins-config-volume\") pod \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\" (UID: \"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f\") " Oct 30 16:10:25.389998 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.389976 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-plugins-config-volume" pod "8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" (UID: "8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:10:25.391579 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.391558 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-kube-api-access-bfl4k" pod "8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" (UID: "8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f"). InnerVolumeSpecName "kube-api-access-bfl4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:10:25.490429 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.490399 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eb79c17-7e58-49b8-a342-52a42af7ae97-kube-api-access-c2c7j\" (UniqueName: \"kubernetes.io/projected/6eb79c17-7e58-49b8-a342-52a42af7ae97-kube-api-access-c2c7j\") pod \"6eb79c17-7e58-49b8-a342-52a42af7ae97\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " Oct 30 16:10:25.490579 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.490531 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eb79c17-7e58-49b8-a342-52a42af7ae97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb79c17-7e58-49b8-a342-52a42af7ae97-plugins-config-volume\") pod \"6eb79c17-7e58-49b8-a342-52a42af7ae97\" (UID: \"6eb79c17-7e58-49b8-a342-52a42af7ae97\") " Oct 30 16:10:25.490899 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.490877 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:25.490971 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.490905 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfl4k\" (UniqueName: \"kubernetes.io/projected/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f-kube-api-access-bfl4k\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:25.490971 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.490931 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb79c17-7e58-49b8-a342-52a42af7ae97-plugins-config-volume" pod "6eb79c17-7e58-49b8-a342-52a42af7ae97" (UID: "6eb79c17-7e58-49b8-a342-52a42af7ae97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:10:25.492382 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.492357 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb79c17-7e58-49b8-a342-52a42af7ae97-kube-api-access-c2c7j" pod "6eb79c17-7e58-49b8-a342-52a42af7ae97" (UID: "6eb79c17-7e58-49b8-a342-52a42af7ae97"). InnerVolumeSpecName "kube-api-access-c2c7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:10:25.559607 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.559548 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:25.572399 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.572380 4401 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:25.592107 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.592085 4401 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eb79c17-7e58-49b8-a342-52a42af7ae97-plugins-config-volume\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:25.592107 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:25.592103 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2c7j\" (UniqueName: \"kubernetes.io/projected/6eb79c17-7e58-49b8-a342-52a42af7ae97-kube-api-access-c2c7j\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:10:26.168719 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.168689 4401 generic.go:356] "Generic (PLEG): container finished" podID="6eb79c17-7e58-49b8-a342-52a42af7ae97" containerID="ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc" exitCode=0 Oct 30 16:10:26.169230 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.168739 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" Oct 30 16:10:26.169230 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.168782 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" event={"ID":"6eb79c17-7e58-49b8-a342-52a42af7ae97","Type":"ContainerDied","Data":"ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc"} Oct 30 16:10:26.169230 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.168861 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb" event={"ID":"6eb79c17-7e58-49b8-a342-52a42af7ae97","Type":"ContainerDied","Data":"1db15a148f380cf359f6afdc42a8a65d9414126fbef48309be9e64732b2460a9"} Oct 30 16:10:26.169230 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.168892 4401 scope.go:122] "RemoveContainer" containerID="ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc" Oct 30 16:10:26.170338 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.170323 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" Oct 30 16:10:26.170418 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.170363 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt" event={"ID":"8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f","Type":"ContainerDied","Data":"2da83ca162850441d7ba1d44315bb1d94f9738d6735c26b399838ac376f2b9f0"} Oct 30 16:10:26.182599 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.182583 4401 scope.go:122] "RemoveContainer" containerID="ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc" Oct 30 16:10:26.182909 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:10:26.182887 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc\": container with ID starting with ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc not found: ID does not exist" containerID="ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc" Oct 30 16:10:26.182974 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.182916 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc"} err="failed to get container status \"ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc\": rpc error: code = NotFound desc = could not find container \"ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc\": container with ID starting with ffeabf2700371451f4d0fd5ee6a2e9db0df8ba1a7d8a6f3739b8c5b94f2a8bcc not found: ID does not exist" Oct 30 16:10:26.182974 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.182932 4401 scope.go:122] "RemoveContainer" containerID="6abd912b1628e27bcda36500bdd393019bf2afaeb3bac7fd06594a4f3c4821ce" Oct 30 16:10:26.184377 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.184360 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:10:26.227257 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.227236 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt"] Oct 30 16:10:26.241405 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.241382 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-df488b-84df57ffbxvfvt"] Oct 30 16:10:26.317675 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.317590 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb"] Oct 30 16:10:26.321130 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:26.321103 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-df488b-7cb855dmlb"] Oct 30 16:10:27.427716 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:27.427685 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eb79c17-7e58-49b8-a342-52a42af7ae97" path="/var/lib/kubelet/pods/6eb79c17-7e58-49b8-a342-52a42af7ae97/volumes" Oct 30 16:10:27.428132 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:27.428079 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" path="/var/lib/kubelet/pods/8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f/volumes" Oct 30 16:10:35.382490 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:35.382445 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:10:50.399869 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:10:50.399811 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:10:52.072848 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:10:52.070677 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:11:02.827285 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:02.827253 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4"] Oct 30 16:11:04.911846 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:04.911788 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:11:08.038858 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:08.038766 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb"] Oct 30 16:11:08.057901 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:08.057866 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5182ff-jqxrb"] Oct 30 16:11:09.431324 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:09.431284 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2cd27cc-3d42-4560-820e-015a5acac66b" path="/var/lib/kubelet/pods/e2cd27cc-3d42-4560-820e-015a5acac66b/volumes" Oct 30 16:11:12.410820 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:12.410778 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:11:13.351160 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:13.351131 4401 scope.go:122] "RemoveContainer" containerID="73b93689b6d57d806d05934e47d2ab70f492dc099df7a74f256c2a5e53b4eb21" Oct 30 16:11:13.360043 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:13.359976 4401 scope.go:122] "RemoveContainer" containerID="8de5f1a36fab89a0e8e5d068b129b65b4359a729665eea71f8a00fcf3007b486" Oct 30 16:11:17.847470 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:17.847403 4401 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" podUID="7c35e2df-7c75-41c7-bfd5-25081925852e" containerName="main" containerID="cri-o://8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96" gracePeriod=45 Oct 30 16:11:18.106766 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.106707 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:11:18.152252 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152223 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c35e2df-7c75-41c7-bfd5-25081925852e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c35e2df-7c75-41c7-bfd5-25081925852e-tls-certs\") pod \"7c35e2df-7c75-41c7-bfd5-25081925852e\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " Oct 30 16:11:18.152403 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152284 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-model-cache\") pod \"7c35e2df-7c75-41c7-bfd5-25081925852e\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " Oct 30 16:11:18.152403 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152354 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-dshm\") pod \"7c35e2df-7c75-41c7-bfd5-25081925852e\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " Oct 30 16:11:18.152521 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152403 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-tmp-dir\") pod \"7c35e2df-7c75-41c7-bfd5-25081925852e\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " Oct 30 16:11:18.152521 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152449 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c35e2df-7c75-41c7-bfd5-25081925852e-kube-api-access-zgl2d\" (UniqueName: \"kubernetes.io/projected/7c35e2df-7c75-41c7-bfd5-25081925852e-kube-api-access-zgl2d\") pod \"7c35e2df-7c75-41c7-bfd5-25081925852e\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " Oct 30 16:11:18.152631 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152543 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-home\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-home\") pod \"7c35e2df-7c75-41c7-bfd5-25081925852e\" (UID: \"7c35e2df-7c75-41c7-bfd5-25081925852e\") " Oct 30 16:11:18.152631 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152575 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-model-cache" pod "7c35e2df-7c75-41c7-bfd5-25081925852e" (UID: "7c35e2df-7c75-41c7-bfd5-25081925852e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:11:18.152792 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152740 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-tmp-dir" pod "7c35e2df-7c75-41c7-bfd5-25081925852e" (UID: "7c35e2df-7c75-41c7-bfd5-25081925852e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:11:18.152953 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152932 4401 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-tmp-dir\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:11:18.153039 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.152958 4401 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-model-cache\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:11:18.153193 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.153153 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-home" pod "7c35e2df-7c75-41c7-bfd5-25081925852e" (UID: "7c35e2df-7c75-41c7-bfd5-25081925852e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:11:18.155427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.155400 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c35e2df-7c75-41c7-bfd5-25081925852e-kube-api-access-zgl2d" pod "7c35e2df-7c75-41c7-bfd5-25081925852e" (UID: "7c35e2df-7c75-41c7-bfd5-25081925852e"). InnerVolumeSpecName "kube-api-access-zgl2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:11:18.155553 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.155434 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-dshm" pod "7c35e2df-7c75-41c7-bfd5-25081925852e" (UID: "7c35e2df-7c75-41c7-bfd5-25081925852e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:11:18.159944 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.159921 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c35e2df-7c75-41c7-bfd5-25081925852e-tls-certs" pod "7c35e2df-7c75-41c7-bfd5-25081925852e" (UID: "7c35e2df-7c75-41c7-bfd5-25081925852e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:11:18.254478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.254448 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgl2d\" (UniqueName: \"kubernetes.io/projected/7c35e2df-7c75-41c7-bfd5-25081925852e-kube-api-access-zgl2d\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:11:18.254478 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.254478 4401 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-home\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:11:18.254663 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.254489 4401 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c35e2df-7c75-41c7-bfd5-25081925852e-tls-certs\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:11:18.254663 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.254499 4401 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c35e2df-7c75-41c7-bfd5-25081925852e-dshm\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:11:18.709441 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.709408 4401 generic.go:356] "Generic (PLEG): container finished" podID="7c35e2df-7c75-41c7-bfd5-25081925852e" containerID="8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96" exitCode=0 Oct 30 16:11:18.709650 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.709449 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" Oct 30 16:11:18.709650 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.709498 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" event={"ID":"7c35e2df-7c75-41c7-bfd5-25081925852e","Type":"ContainerDied","Data":"8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96"} Oct 30 16:11:18.709650 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.709541 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4" event={"ID":"7c35e2df-7c75-41c7-bfd5-25081925852e","Type":"ContainerDied","Data":"d6380236c210e33d0b089a8555d66f996663dca35db092c6552d4c1469a6fece"} Oct 30 16:11:18.709650 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.709561 4401 scope.go:122] "RemoveContainer" containerID="8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96" Oct 30 16:11:18.719470 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.719454 4401 scope.go:122] "RemoveContainer" containerID="8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96" Oct 30 16:11:18.719735 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:11:18.719715 4401 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96\": container with ID starting with 8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96 not found: ID does not exist" containerID="8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96" Oct 30 16:11:18.719792 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.719747 4401 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96"} err="failed to get container status \"8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96\": rpc error: code = NotFound desc = could not find container \"8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96\": container with ID starting with 8f0a2c7d44465c7d45a6691233e684ccdfb98bee805d473ec626c9818d488f96 not found: ID does not exist" Oct 30 16:11:18.766996 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.766967 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4"] Oct 30 16:11:18.780967 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:18.780940 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3c05ee64-kserve-5c9c94748f-xvkm4"] Oct 30 16:11:19.426322 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:19.426288 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c35e2df-7c75-41c7-bfd5-25081925852e" path="/var/lib/kubelet/pods/7c35e2df-7c75-41c7-bfd5-25081925852e/volumes" Oct 30 16:11:20.878428 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:20.878394 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:11:22.073069 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:11:22.072944 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:11:26.945438 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:26.945387 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:11:44.689769 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:44.689731 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:11:51.966009 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:11:51.965973 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:11:54.611745 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:11:54.611667 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:12:02.880463 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:02.880426 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:12:05.467848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.467030 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-trg66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:12:05.467848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.467110 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-trg66" podUID="8c6639c2-1eac-4716-8a14-cd33938b1c67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:12:05.509859 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.508955 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-szdqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:12:05.509859 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.509019 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-szdqp" podUID="636add60-0495-48b7-bd72-1cebe229fa0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:12:05.626979 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.626929 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:12:05.627304 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.627255 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-rsp9f" podUID="6cd43595-43d2-4f68-85bb-fea131b69e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:12:05.733073 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.732961 4401 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-9dccdc9b7-862j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:12:05.733371 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:05.733323 4401 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-9dccdc9b7-862j7" podUID="672a1ec4-2d4f-4a6f-8e4a-369b6be336c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:12:09.683665 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.683633 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt"] Oct 30 16:12:09.684105 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684091 4401 cpu_manager.go:414] "containerMap: removing container" podUID="7c35e2df-7c75-41c7-bfd5-25081925852e" containerName="main" Oct 30 16:12:09.684151 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684106 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c35e2df-7c75-41c7-bfd5-25081925852e" containerName="main" Oct 30 16:12:09.684151 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684117 4401 cpu_manager.go:414] "containerMap: removing container" podUID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" containerName="payload-processing" Oct 30 16:12:09.684151 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684122 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" containerName="payload-processing" Oct 30 16:12:09.684151 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684139 4401 cpu_manager.go:414] "containerMap: removing container" podUID="6eb79c17-7e58-49b8-a342-52a42af7ae97" containerName="payload-pre-processing" Oct 30 16:12:09.684151 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684145 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eb79c17-7e58-49b8-a342-52a42af7ae97" containerName="payload-pre-processing" Oct 30 16:12:09.684304 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684206 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c35e2df-7c75-41c7-bfd5-25081925852e" containerName="main" Oct 30 16:12:09.684304 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684216 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="8536cd6c-0fb0-470c-9a6d-d38ca7d5a35f" containerName="payload-processing" Oct 30 16:12:09.684304 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.684224 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eb79c17-7e58-49b8-a342-52a42af7ae97" containerName="payload-pre-processing" Oct 30 16:12:09.687253 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.687238 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.704778 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.704753 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt"] Oct 30 16:12:09.716791 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.716767 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:12:09.716791 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.716773 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:12:09.718131 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.718116 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:12:09.718202 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.718152 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:12:09.778590 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.778564 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmm6d\" (UniqueName: \"kubernetes.io/projected/59da2cac-df09-4999-9625-de56a0458fd4-kube-api-access-rmm6d\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.778739 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.778711 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.778868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.778847 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.879412 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.879384 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.879525 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.879420 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmm6d\" (UniqueName: \"kubernetes.io/projected/59da2cac-df09-4999-9625-de56a0458fd4-kube-api-access-rmm6d\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.879525 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.879497 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.880074 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.880051 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.881722 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.881702 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.905212 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.905189 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmm6d\" (UniqueName: \"kubernetes.io/projected/59da2cac-df09-4999-9625-de56a0458fd4-kube-api-access-rmm6d\") pod \"maas-api-revoke-sub-f03629db-jqbkt\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:09.999494 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:09.999441 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:10.301643 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:10.301612 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt"] Oct 30 16:12:10.302803 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:12:10.302780 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59da2cac_df09_4999_9625_de56a0458fd4.slice/crio-1d6454bd8a285fad7f3a26d1a00ebdf37e867d97ebae5d2953013736ece5c98d WatchSource:0}: Error finding container 1d6454bd8a285fad7f3a26d1a00ebdf37e867d97ebae5d2953013736ece5c98d: Status 404 returned error can't find the container with id 1d6454bd8a285fad7f3a26d1a00ebdf37e867d97ebae5d2953013736ece5c98d Oct 30 16:12:11.248913 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:11.248880 4401 generic.go:356] "Generic (PLEG): container finished" podID="59da2cac-df09-4999-9625-de56a0458fd4" containerID="2c94322f877b94074b3994a17d11d67016a63e2b585e3909b69cbfd59ee827a4" exitCode=0 Oct 30 16:12:11.249377 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:11.248971 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" event={"ID":"59da2cac-df09-4999-9625-de56a0458fd4","Type":"ContainerDied","Data":"2c94322f877b94074b3994a17d11d67016a63e2b585e3909b69cbfd59ee827a4"} Oct 30 16:12:11.249377 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:11.249020 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" event={"ID":"59da2cac-df09-4999-9625-de56a0458fd4","Type":"ContainerStarted","Data":"1d6454bd8a285fad7f3a26d1a00ebdf37e867d97ebae5d2953013736ece5c98d"} Oct 30 16:12:12.394881 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.394857 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:12.503669 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.503638 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-openshift-ca\") pod \"59da2cac-df09-4999-9625-de56a0458fd4\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " Oct 30 16:12:12.503843 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.503736 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59da2cac-df09-4999-9625-de56a0458fd4-kube-api-access-rmm6d\" (UniqueName: \"kubernetes.io/projected/59da2cac-df09-4999-9625-de56a0458fd4-kube-api-access-rmm6d\") pod \"59da2cac-df09-4999-9625-de56a0458fd4\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " Oct 30 16:12:12.503843 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.503790 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-xks-ca\") pod \"59da2cac-df09-4999-9625-de56a0458fd4\" (UID: \"59da2cac-df09-4999-9625-de56a0458fd4\") " Oct 30 16:12:12.504040 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.504013 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-openshift-ca" pod "59da2cac-df09-4999-9625-de56a0458fd4" (UID: "59da2cac-df09-4999-9625-de56a0458fd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:12:12.505840 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.505776 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-xks-ca" pod "59da2cac-df09-4999-9625-de56a0458fd4" (UID: "59da2cac-df09-4999-9625-de56a0458fd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:12:12.505923 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.505856 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59da2cac-df09-4999-9625-de56a0458fd4-kube-api-access-rmm6d" pod "59da2cac-df09-4999-9625-de56a0458fd4" (UID: "59da2cac-df09-4999-9625-de56a0458fd4"). InnerVolumeSpecName "kube-api-access-rmm6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:12:12.605087 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.605060 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmm6d\" (UniqueName: \"kubernetes.io/projected/59da2cac-df09-4999-9625-de56a0458fd4-kube-api-access-rmm6d\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:12.605087 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.605083 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:12.605087 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:12.605094 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59da2cac-df09-4999-9625-de56a0458fd4-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:13.261276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:13.261237 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" event={"ID":"59da2cac-df09-4999-9625-de56a0458fd4","Type":"ContainerDied","Data":"1d6454bd8a285fad7f3a26d1a00ebdf37e867d97ebae5d2953013736ece5c98d"} Oct 30 16:12:13.261276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:13.261268 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-jqbkt" Oct 30 16:12:13.261276 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:13.261287 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d6454bd8a285fad7f3a26d1a00ebdf37e867d97ebae5d2953013736ece5c98d" Oct 30 16:12:13.392676 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:13.392642 4401 scope.go:122] "RemoveContainer" containerID="6908cd9180ee5401d1974e8c25bc198cc5213f589df3f6d59b770e84a76fc12f" Oct 30 16:12:16.499330 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:16.499295 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:12:21.970769 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:12:21.970733 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:12:22.094555 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.094512 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx"] Oct 30 16:12:22.095014 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.094996 4401 cpu_manager.go:414] "containerMap: removing container" podUID="59da2cac-df09-4999-9625-de56a0458fd4" containerName="revoke-keys" Oct 30 16:12:22.095014 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.095011 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59da2cac-df09-4999-9625-de56a0458fd4" containerName="revoke-keys" Oct 30 16:12:22.095233 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.095114 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="59da2cac-df09-4999-9625-de56a0458fd4" containerName="revoke-keys" Oct 30 16:12:22.100051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.100022 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.117036 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.117009 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx"] Oct 30 16:12:22.128390 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.128362 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:12:22.128546 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.128484 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:12:22.128546 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.128484 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:12:22.130950 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.130927 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:12:22.145728 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.145699 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.145884 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.145754 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.145884 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.145787 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8vvl\" (UniqueName: \"kubernetes.io/projected/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-kube-api-access-p8vvl\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.246647 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.246549 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.246647 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.246599 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8vvl\" (UniqueName: \"kubernetes.io/projected/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-kube-api-access-p8vvl\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.246989 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.246670 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.247350 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.247328 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.248989 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.248967 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.272306 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.272272 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8vvl\" (UniqueName: \"kubernetes.io/projected/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-kube-api-access-p8vvl\") pod \"maas-api-revoke-sub-0165d7d2-kgdxx\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.413506 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.413473 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:22.715465 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:22.715440 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx"] Oct 30 16:12:22.716470 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:12:22.716445 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c7ea7d_eb55_48fc_8793_c915b9d4ae04.slice/crio-db5489001b3b56f2fd46c3efb9b418b93f6d19841e4a2b73c91f05c7b002ca6f WatchSource:0}: Error finding container db5489001b3b56f2fd46c3efb9b418b93f6d19841e4a2b73c91f05c7b002ca6f: Status 404 returned error can't find the container with id db5489001b3b56f2fd46c3efb9b418b93f6d19841e4a2b73c91f05c7b002ca6f Oct 30 16:12:23.341583 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:23.341498 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" event={"ID":"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04","Type":"ContainerStarted","Data":"a1718e8cfe05d17ea925cba42a7af115d16491bd054d721c75ea18dc3c59e8ad"} Oct 30 16:12:23.341583 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:23.341563 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" event={"ID":"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04","Type":"ContainerStarted","Data":"db5489001b3b56f2fd46c3efb9b418b93f6d19841e4a2b73c91f05c7b002ca6f"} Oct 30 16:12:23.402388 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:23.402331 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" podStartSLOduration=1.402318973 podStartE2EDuration="1.402318973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:12:23.387494684 +0000 UTC m=+2052.086767840" watchObservedRunningTime="2026-10-30 16:12:23.402318973 +0000 UTC m=+2052.101592130" Oct 30 16:12:24.349318 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:24.349287 4401 generic.go:356] "Generic (PLEG): container finished" podID="f2c7ea7d-eb55-48fc-8793-c915b9d4ae04" containerID="a1718e8cfe05d17ea925cba42a7af115d16491bd054d721c75ea18dc3c59e8ad" exitCode=0 Oct 30 16:12:24.349725 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:24.349327 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" event={"ID":"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04","Type":"ContainerDied","Data":"a1718e8cfe05d17ea925cba42a7af115d16491bd054d721c75ea18dc3c59e8ad"} Oct 30 16:12:25.489409 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.489386 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:25.576548 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.576510 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-kube-api-access-p8vvl\" (UniqueName: \"kubernetes.io/projected/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-kube-api-access-p8vvl\") pod \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " Oct 30 16:12:25.576755 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.576605 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-openshift-ca\") pod \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " Oct 30 16:12:25.576755 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.576646 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-xks-ca\") pod \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\" (UID: \"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04\") " Oct 30 16:12:25.576996 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.576971 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-openshift-ca" pod "f2c7ea7d-eb55-48fc-8793-c915b9d4ae04" (UID: "f2c7ea7d-eb55-48fc-8793-c915b9d4ae04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:12:25.578666 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.578643 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-kube-api-access-p8vvl" pod "f2c7ea7d-eb55-48fc-8793-c915b9d4ae04" (UID: "f2c7ea7d-eb55-48fc-8793-c915b9d4ae04"). InnerVolumeSpecName "kube-api-access-p8vvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:12:25.578748 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.578722 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-xks-ca" pod "f2c7ea7d-eb55-48fc-8793-c915b9d4ae04" (UID: "f2c7ea7d-eb55-48fc-8793-c915b9d4ae04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:12:25.677880 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.677801 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8vvl\" (UniqueName: \"kubernetes.io/projected/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-kube-api-access-p8vvl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:25.677880 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.677856 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:25.677880 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:25.677867 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2c7ea7d-eb55-48fc-8793-c915b9d4ae04-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:26.362310 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:26.362278 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" Oct 30 16:12:26.362471 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:26.362309 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-kgdxx" event={"ID":"f2c7ea7d-eb55-48fc-8793-c915b9d4ae04","Type":"ContainerDied","Data":"db5489001b3b56f2fd46c3efb9b418b93f6d19841e4a2b73c91f05c7b002ca6f"} Oct 30 16:12:26.362471 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:26.362360 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db5489001b3b56f2fd46c3efb9b418b93f6d19841e4a2b73c91f05c7b002ca6f" Oct 30 16:12:33.131920 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.131885 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8"] Oct 30 16:12:33.132325 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.132313 4401 cpu_manager.go:414] "containerMap: removing container" podUID="f2c7ea7d-eb55-48fc-8793-c915b9d4ae04" containerName="revoke-keys" Oct 30 16:12:33.132386 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.132326 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c7ea7d-eb55-48fc-8793-c915b9d4ae04" containerName="revoke-keys" Oct 30 16:12:33.132427 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.132414 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c7ea7d-eb55-48fc-8793-c915b9d4ae04" containerName="revoke-keys" Oct 30 16:12:33.135928 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.135905 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.148378 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.148335 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh8zl\" (UniqueName: \"kubernetes.io/projected/89cfc4ce-ffe0-42ab-904b-0830d944432a-kube-api-access-fh8zl\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.148531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.148446 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.148531 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.148473 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.154368 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.154340 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8"] Oct 30 16:12:33.164939 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.164908 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:12:33.164939 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.164935 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:12:33.165785 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.165001 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:12:33.167032 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.167012 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:12:33.249857 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.249795 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.250058 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.249862 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.250058 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.249997 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh8zl\" (UniqueName: \"kubernetes.io/projected/89cfc4ce-ffe0-42ab-904b-0830d944432a-kube-api-access-fh8zl\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.250458 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.250436 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.252219 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.252198 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.275373 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.275348 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh8zl\" (UniqueName: \"kubernetes.io/projected/89cfc4ce-ffe0-42ab-904b-0830d944432a-kube-api-access-fh8zl\") pod \"maas-api-revoke-sub-6b19338f-xgpn8\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.449752 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.449674 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:33.754223 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:33.754198 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8"] Oct 30 16:12:33.755701 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:12:33.755674 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cfc4ce_ffe0_42ab_904b_0830d944432a.slice/crio-8080b0eb107769c0d184db7f6bdf825699369752eb2d27411bf4c07f5387432c WatchSource:0}: Error finding container 8080b0eb107769c0d184db7f6bdf825699369752eb2d27411bf4c07f5387432c: Status 404 returned error can't find the container with id 8080b0eb107769c0d184db7f6bdf825699369752eb2d27411bf4c07f5387432c Oct 30 16:12:34.414486 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:34.414447 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" event={"ID":"89cfc4ce-ffe0-42ab-904b-0830d944432a","Type":"ContainerStarted","Data":"28ee6c4a8dc962122b1c42a169638084b8be26a5ff222e3238fbc83ee3a36c27"} Oct 30 16:12:34.414885 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:34.414512 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" event={"ID":"89cfc4ce-ffe0-42ab-904b-0830d944432a","Type":"ContainerStarted","Data":"8080b0eb107769c0d184db7f6bdf825699369752eb2d27411bf4c07f5387432c"} Oct 30 16:12:34.461041 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:34.460997 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" podStartSLOduration=1.460984249 podStartE2EDuration="1.460984249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:12:34.458937438 +0000 UTC m=+2063.158210594" watchObservedRunningTime="2026-10-30 16:12:34.460984249 +0000 UTC m=+2063.160257406" Oct 30 16:12:35.422628 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:35.422591 4401 generic.go:356] "Generic (PLEG): container finished" podID="89cfc4ce-ffe0-42ab-904b-0830d944432a" containerID="28ee6c4a8dc962122b1c42a169638084b8be26a5ff222e3238fbc83ee3a36c27" exitCode=0 Oct 30 16:12:35.433539 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:35.433508 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" event={"ID":"89cfc4ce-ffe0-42ab-904b-0830d944432a","Type":"ContainerDied","Data":"28ee6c4a8dc962122b1c42a169638084b8be26a5ff222e3238fbc83ee3a36c27"} Oct 30 16:12:36.568117 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.568095 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:36.687070 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.686986 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-openshift-ca\") pod \"89cfc4ce-ffe0-42ab-904b-0830d944432a\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " Oct 30 16:12:36.687226 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.687069 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89cfc4ce-ffe0-42ab-904b-0830d944432a-kube-api-access-fh8zl\" (UniqueName: \"kubernetes.io/projected/89cfc4ce-ffe0-42ab-904b-0830d944432a-kube-api-access-fh8zl\") pod \"89cfc4ce-ffe0-42ab-904b-0830d944432a\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " Oct 30 16:12:36.687226 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.687214 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-xks-ca\") pod \"89cfc4ce-ffe0-42ab-904b-0830d944432a\" (UID: \"89cfc4ce-ffe0-42ab-904b-0830d944432a\") " Oct 30 16:12:36.687386 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.687357 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-openshift-ca" pod "89cfc4ce-ffe0-42ab-904b-0830d944432a" (UID: "89cfc4ce-ffe0-42ab-904b-0830d944432a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:12:36.687549 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.687531 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-openshift-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:36.689176 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.689153 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89cfc4ce-ffe0-42ab-904b-0830d944432a-kube-api-access-fh8zl" pod "89cfc4ce-ffe0-42ab-904b-0830d944432a" (UID: "89cfc4ce-ffe0-42ab-904b-0830d944432a"). InnerVolumeSpecName "kube-api-access-fh8zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:12:36.689176 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.689161 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-xks-ca" pod "89cfc4ce-ffe0-42ab-904b-0830d944432a" (UID: "89cfc4ce-ffe0-42ab-904b-0830d944432a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:12:36.788617 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.788584 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89cfc4ce-ffe0-42ab-904b-0830d944432a-maas-api-xks-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:36.788617 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:36.788614 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh8zl\" (UniqueName: \"kubernetes.io/projected/89cfc4ce-ffe0-42ab-904b-0830d944432a-kube-api-access-fh8zl\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:12:37.435809 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:37.435785 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" Oct 30 16:12:37.435966 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:37.435855 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-xgpn8" event={"ID":"89cfc4ce-ffe0-42ab-904b-0830d944432a","Type":"ContainerDied","Data":"8080b0eb107769c0d184db7f6bdf825699369752eb2d27411bf4c07f5387432c"} Oct 30 16:12:37.435966 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:37.435897 4401 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8080b0eb107769c0d184db7f6bdf825699369752eb2d27411bf4c07f5387432c" Oct 30 16:12:44.482658 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:44.482627 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-2qndw"] Oct 30 16:12:51.966446 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:12:51.966408 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:12:55.210930 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:55.210882 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d48c9567-8qzkb_8803c379-9a94-4756-adae-baa7172b3212/manager/0.log" Oct 30 16:12:56.217230 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:56.217200 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5964675dcd-kxw7t_222d4524-f967-4258-bb9c-9252db660785/manager/0.log" Oct 30 16:12:57.647969 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:57.647938 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7_ce734ad7-04f7-4520-a0f7-3710d393736a/copy-manifests/0.log" Oct 30 16:12:57.680602 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:57.680577 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7_ce734ad7-04f7-4520-a0f7-3710d393736a/manager/0.log" Oct 30 16:12:59.079289 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:59.079258 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kgdxx_f2c7ea7d-eb55-48fc-8793-c915b9d4ae04/revoke-keys/0.log" Oct 30 16:12:59.456733 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:59.456706 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3bd20fdf-q524p_ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc/revoke-keys/0.log" Oct 30 16:12:59.995609 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:12:59.995568 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xgpn8_89cfc4ce-ffe0-42ab-904b-0830d944432a/revoke-keys/0.log" Oct 30 16:13:00.363098 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:00.363069 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-qwqzm_ab32563d-f5bf-4a7b-80c6-84d6d6c04e53/revoke-keys/0.log" Oct 30 16:13:00.547122 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:00.547091 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-jqbkt_59da2cac-df09-4999-9625-de56a0458fd4/revoke-keys/0.log" Oct 30 16:13:02.539663 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:02.539627 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-2rdzt_f4102670-ceac-42d4-8a3c-b685f5d4d43c/manager/0.log" Oct 30 16:13:03.425575 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:03.425540 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2qndw_7f6e42ec-f48a-4003-abb8-ce3d06af947a/limitador/0.log" Oct 30 16:13:03.628377 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:03.628350 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-dx5wm_1b0b1350-4d9c-4012-8761-ddccf2e06879/manager/0.log" Oct 30 16:13:04.415751 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:04.415717 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f29t5m_75b74b6b-5966-4dbb-9e14-92c95de9d211/istio-proxy/0.log" Oct 30 16:13:04.621890 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:04.621861 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jndpr_afd4958a-0741-47b7-a199-9e51f9cf2e9a/discovery/0.log" Oct 30 16:13:05.035445 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:05.035414 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f6967bb6d-fjrbx_1d1b87c2-0b2c-437b-95db-76f18662d056/kube-auth-proxy/0.log" Oct 30 16:13:05.233909 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:05.233884 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-862j7_672a1ec4-2d4f-4a6f-8e4a-369b6be336c7/istio-proxy/0.log" Oct 30 16:13:06.465418 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:06.465389 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-rsp9f_6cd43595-43d2-4f68-85bb-fea131b69e72/istio-proxy/0.log" Oct 30 16:13:06.891801 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:06.891771 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-szdqp_636add60-0495-48b7-bd72-1cebe229fa0c/istio-proxy/0.log" Oct 30 16:13:07.103785 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:07.103748 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-trg66_8c6639c2-1eac-4716-8a14-cd33938b1c67/istio-proxy/0.log" Oct 30 16:13:07.300424 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:07.300343 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6tt6h_0567028d-7000-45f9-ab89-c4524254517c/payload-pre-processing/0.log" Oct 30 16:13:07.514497 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:07.514466 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bfqbb_b1a28f78-9ced-4190-9c1f-413f206fd8c5/payload-processing/0.log" Oct 30 16:13:08.874282 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:08.874249 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8spw6_b4f4f6ca-7df0-428a-8b91-9e9301a154b4/main/0.log" Oct 30 16:13:09.286165 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:09.286077 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z_3b8a62e2-b40a-4467-a128-46d2ba1d2fcc/main/0.log" Oct 30 16:13:13.417811 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:13.417777 4401 scope.go:122] "RemoveContainer" containerID="f6549e681c054bffad2816e7607a4f332445183eee3c43109a84d58b4d456d86" Oct 30 16:13:19.835630 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:19.835596 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-54d48c9567-8qzkb_8803c379-9a94-4756-adae-baa7172b3212/manager/0.log" Oct 30 16:13:20.719522 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:20.719476 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5964675dcd-kxw7t_222d4524-f967-4258-bb9c-9252db660785/manager/0.log" Oct 30 16:13:21.965598 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:13:21.965556 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:13:22.129241 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:22.129214 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7_ce734ad7-04f7-4520-a0f7-3710d393736a/copy-manifests/0.log" Oct 30 16:13:22.159246 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:22.159219 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5b7c6fc8bc-lj4g7_ce734ad7-04f7-4520-a0f7-3710d393736a/manager/0.log" Oct 30 16:13:23.355263 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:23.355232 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-kgdxx_f2c7ea7d-eb55-48fc-8793-c915b9d4ae04/revoke-keys/0.log" Oct 30 16:13:23.737657 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:23.737574 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3bd20fdf-q524p_ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc/revoke-keys/0.log" Oct 30 16:13:24.312582 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:24.312510 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-xgpn8_89cfc4ce-ffe0-42ab-904b-0830d944432a/revoke-keys/0.log" Oct 30 16:13:24.681290 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:24.681261 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-qwqzm_ab32563d-f5bf-4a7b-80c6-84d6d6c04e53/revoke-keys/0.log" Oct 30 16:13:24.886515 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:24.886486 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-jqbkt_59da2cac-df09-4999-9625-de56a0458fd4/revoke-keys/0.log" Oct 30 16:13:26.423707 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:26.423675 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-2rdzt_f4102670-ceac-42d4-8a3c-b685f5d4d43c/manager/0.log" Oct 30 16:13:27.296302 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:27.296272 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2qndw_7f6e42ec-f48a-4003-abb8-ce3d06af947a/limitador/0.log" Oct 30 16:13:27.493331 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:27.493297 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-dx5wm_1b0b1350-4d9c-4012-8761-ddccf2e06879/manager/0.log" Oct 30 16:13:28.108133 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:28.108106 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f29t5m_75b74b6b-5966-4dbb-9e14-92c95de9d211/istio-proxy/0.log" Oct 30 16:13:28.306482 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:28.306455 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jndpr_afd4958a-0741-47b7-a199-9e51f9cf2e9a/discovery/0.log" Oct 30 16:13:28.692035 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:28.692002 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f6967bb6d-fjrbx_1d1b87c2-0b2c-437b-95db-76f18662d056/kube-auth-proxy/0.log" Oct 30 16:13:28.887088 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:28.887059 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-862j7_672a1ec4-2d4f-4a6f-8e4a-369b6be336c7/istio-proxy/0.log" Oct 30 16:13:30.122003 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:30.121973 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-rsp9f_6cd43595-43d2-4f68-85bb-fea131b69e72/istio-proxy/0.log" Oct 30 16:13:30.542207 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:30.542129 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-szdqp_636add60-0495-48b7-bd72-1cebe229fa0c/istio-proxy/0.log" Oct 30 16:13:30.749881 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:30.749846 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-trg66_8c6639c2-1eac-4716-8a14-cd33938b1c67/istio-proxy/0.log" Oct 30 16:13:30.946204 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:30.946175 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6tt6h_0567028d-7000-45f9-ab89-c4524254517c/payload-pre-processing/0.log" Oct 30 16:13:31.136165 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:31.136136 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bfqbb_b1a28f78-9ced-4190-9c1f-413f206fd8c5/payload-processing/0.log" Oct 30 16:13:32.238411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:32.238385 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8spw6_b4f4f6ca-7df0-428a-8b91-9e9301a154b4/main/0.log" Oct 30 16:13:32.611485 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:32.611449 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8phq6z_3b8a62e2-b40a-4467-a128-46d2ba1d2fcc/main/0.log" Oct 30 16:13:33.030711 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:33.030625 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm"] Oct 30 16:13:33.046673 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:33.046638 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-qwqzm"] Oct 30 16:13:33.426908 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:13:33.426877 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab32563d-f5bf-4a7b-80c6-84d6d6c04e53" path="/var/lib/kubelet/pods/ab32563d-f5bf-4a7b-80c6-84d6d6c04e53/volumes" Oct 30 16:13:51.966212 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:13:51.966170 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:14:09.380757 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:09.380720 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5964675dcd-kxw7t_222d4524-f967-4258-bb9c-9252db660785/manager/0.log" Oct 30 16:14:09.686411 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:09.686332 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5964675dcd-kxw7t_222d4524-f967-4258-bb9c-9252db660785/manager/0.log" Oct 30 16:14:10.451506 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:10.451480 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bfqbb_b1a28f78-9ced-4190-9c1f-413f206fd8c5/payload-processing/0.log" Oct 30 16:14:10.652670 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:10.652640 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bfqbb_b1a28f78-9ced-4190-9c1f-413f206fd8c5/payload-processing/0.log" Oct 30 16:14:21.965545 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:14:21.965496 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:14:42.030887 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:42.030853 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p"] Oct 30 16:14:42.033477 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:42.033456 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bd20fdf-q524p"] Oct 30 16:14:43.426442 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:43.426406 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc" path="/var/lib/kubelet/pods/ce3dcbad-d699-4d0b-b7a8-8c509b10b5bc/volumes" Oct 30 16:14:51.966195 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:14:51.966150 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:14:55.266523 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:55.266450 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5964675dcd-kxw7t_222d4524-f967-4258-bb9c-9252db660785/manager/0.log" Oct 30 16:14:55.562393 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:55.562312 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5964675dcd-kxw7t_222d4524-f967-4258-bb9c-9252db660785/manager/0.log" Oct 30 16:14:56.293842 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.293804 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bfqbb_b1a28f78-9ced-4190-9c1f-413f206fd8c5/payload-processing/0.log" Oct 30 16:14:56.480620 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.480588 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bfqbb_b1a28f78-9ced-4190-9c1f-413f206fd8c5/payload-processing/0.log" Oct 30 16:14:56.829883 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.829808 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vkvmm/must-gather-bv65x"] Oct 30 16:14:56.830383 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.830365 4401 cpu_manager.go:414] "containerMap: removing container" podUID="89cfc4ce-ffe0-42ab-904b-0830d944432a" containerName="revoke-keys" Oct 30 16:14:56.830445 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.830384 4401 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89cfc4ce-ffe0-42ab-904b-0830d944432a" containerName="revoke-keys" Oct 30 16:14:56.830482 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.830469 4401 memory_manager.go:365] "RemoveStaleState removing state" podUID="89cfc4ce-ffe0-42ab-904b-0830d944432a" containerName="revoke-keys" Oct 30 16:14:56.834116 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.834095 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:56.865839 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.865794 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vkvmm\"/\"openshift-service-ca.crt\"" Oct 30 16:14:56.865839 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.865817 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vkvmm\"/\"kube-root-ca.crt\"" Oct 30 16:14:56.866017 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.865846 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vkvmm\"/\"default-dockercfg-w6wsh\"" Oct 30 16:14:56.871982 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.871962 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vkvmm/must-gather-bv65x"] Oct 30 16:14:56.945653 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.945618 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4-must-gather-output\") pod \"must-gather-bv65x\" (UID: \"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4\") " pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:56.945859 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:56.945737 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jds\" (UniqueName: \"kubernetes.io/projected/2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4-kube-api-access-25jds\") pod \"must-gather-bv65x\" (UID: \"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4\") " pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:57.047022 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:57.046968 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jds\" (UniqueName: \"kubernetes.io/projected/2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4-kube-api-access-25jds\") pod \"must-gather-bv65x\" (UID: \"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4\") " pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:57.047213 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:57.047109 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4-must-gather-output\") pod \"must-gather-bv65x\" (UID: \"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4\") " pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:57.047454 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:57.047432 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4-must-gather-output\") pod \"must-gather-bv65x\" (UID: \"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4\") " pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:57.077262 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:57.077234 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jds\" (UniqueName: \"kubernetes.io/projected/2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4-kube-api-access-25jds\") pod \"must-gather-bv65x\" (UID: \"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4\") " pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:57.148413 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:57.148388 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkvmm/must-gather-bv65x" Oct 30 16:14:57.450539 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:57.450507 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vkvmm/must-gather-bv65x"] Oct 30 16:14:57.451744 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:14:57.451721 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2562e3c9_f3c4_40d3_9836_dbdc14b8b7f4.slice/crio-d232094128c1b9c90c0de4282ee1144a06959b7615366fc3422f6966cf9f1edc WatchSource:0}: Error finding container d232094128c1b9c90c0de4282ee1144a06959b7615366fc3422f6966cf9f1edc: Status 404 returned error can't find the container with id d232094128c1b9c90c0de4282ee1144a06959b7615366fc3422f6966cf9f1edc Oct 30 16:14:57.453466 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:57.453446 4401 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:14:58.268866 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:58.268837 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkvmm/must-gather-bv65x" event={"ID":"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4","Type":"ContainerStarted","Data":"d232094128c1b9c90c0de4282ee1144a06959b7615366fc3422f6966cf9f1edc"} Oct 30 16:14:59.279210 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:59.279175 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkvmm/must-gather-bv65x" event={"ID":"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4","Type":"ContainerStarted","Data":"34ce89129cbc3af70c084766634e2843c7a23ed10693f649984738d11c8f2740"} Oct 30 16:14:59.279591 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:59.279220 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkvmm/must-gather-bv65x" event={"ID":"2562e3c9-f3c4-40d3-9836-dbdc14b8b7f4","Type":"ContainerStarted","Data":"abdd19de33219e54a7bf3964ea5d261e60ca67ffacb29526222e94644f8309dd"} Oct 30 16:14:59.324497 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:14:59.324442 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vkvmm/must-gather-bv65x" podStartSLOduration=2.530669943 podStartE2EDuration="3.324427379s" totalImagesPullingTime="793.757436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:14:59.322446865 +0000 UTC m=+2208.021720022" watchObservedRunningTime="2026-10-30 16:14:59.324427379 +0000 UTC m=+2208.023700555" Oct 30 16:15:00.150854 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.150801 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv"] Oct 30 16:15:00.156665 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.156643 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.173493 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.173462 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv"] Oct 30 16:15:00.184048 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.184024 4401 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wsxv7\"" Oct 30 16:15:00.215881 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.215850 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:15:00.216067 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.215913 4401 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:15:00.288305 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.288272 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17d6246-6f45-4250-83ab-0002f72feb88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889615-ds4mv\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.288690 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.288310 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/f17d6246-6f45-4250-83ab-0002f72feb88-kube-api-access-xfrtv\") pod \"maas-api-key-cleanup-29889615-ds4mv\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.391733 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.391686 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17d6246-6f45-4250-83ab-0002f72feb88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889615-ds4mv\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.391943 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.391742 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/f17d6246-6f45-4250-83ab-0002f72feb88-kube-api-access-xfrtv\") pod \"maas-api-key-cleanup-29889615-ds4mv\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.392392 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.392366 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17d6246-6f45-4250-83ab-0002f72feb88-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889615-ds4mv\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.422070 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.422000 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/f17d6246-6f45-4250-83ab-0002f72feb88-kube-api-access-xfrtv\") pod \"maas-api-key-cleanup-29889615-ds4mv\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.480470 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.480438 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:00.800580 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:00.800551 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv"] Oct 30 16:15:00.803059 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:15:00.803030 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17d6246_6f45_4250_83ab_0002f72feb88.slice/crio-7673a2f8fd0e772f20da8eff95c42f1345644ed90130320c19a7de15bd67ad29 WatchSource:0}: Error finding container 7673a2f8fd0e772f20da8eff95c42f1345644ed90130320c19a7de15bd67ad29: Status 404 returned error can't find the container with id 7673a2f8fd0e772f20da8eff95c42f1345644ed90130320c19a7de15bd67ad29 Oct 30 16:15:01.296752 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:01.295542 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" event={"ID":"f17d6246-6f45-4250-83ab-0002f72feb88","Type":"ContainerStarted","Data":"020308e90522c04709a49a9759a9be5ec2385597ac60ea1663562aaa83231082"} Oct 30 16:15:01.296752 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:01.295613 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" event={"ID":"f17d6246-6f45-4250-83ab-0002f72feb88","Type":"ContainerStarted","Data":"7673a2f8fd0e772f20da8eff95c42f1345644ed90130320c19a7de15bd67ad29"} Oct 30 16:15:01.343303 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:01.340489 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" podStartSLOduration=1.340469014 podStartE2EDuration="1.340469014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:15:01.339567911 +0000 UTC m=+2210.038841068" watchObservedRunningTime="2026-10-30 16:15:01.340469014 +0000 UTC m=+2210.039742174" Oct 30 16:15:01.491359 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:01.491312 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8r6df_5a9c1136-39de-45c4-ba36-d8a4b524f252/global-pull-secret-syncer/0.log" Oct 30 16:15:02.308354 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:02.308311 4401 generic.go:356] "Generic (PLEG): container finished" podID="f17d6246-6f45-4250-83ab-0002f72feb88" containerID="020308e90522c04709a49a9759a9be5ec2385597ac60ea1663562aaa83231082" exitCode=60 Oct 30 16:15:02.308781 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:02.308537 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" event={"ID":"f17d6246-6f45-4250-83ab-0002f72feb88","Type":"ContainerDied","Data":"020308e90522c04709a49a9759a9be5ec2385597ac60ea1663562aaa83231082"} Oct 30 16:15:02.309007 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:02.308984 4401 scope.go:122] "RemoveContainer" containerID="020308e90522c04709a49a9759a9be5ec2385597ac60ea1663562aaa83231082" Oct 30 16:15:02.626845 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:02.614783 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-scbpb_3b3e40fa-3cc7-4e68-b204-98ff6f6e9d07/konnectivity-agent/0.log" Oct 30 16:15:03.042408 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:03.042327 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-222.ec2.internal_e2bab30e2cbbbd657a1d2e615d7ce89c/haproxy/0.log" Oct 30 16:15:03.393410 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:03.393234 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" event={"ID":"f17d6246-6f45-4250-83ab-0002f72feb88","Type":"ContainerStarted","Data":"690067675f69ca31141d2ae68cd6dd2cf2f259ac9da795d3fe612dec3a36dc91"} Oct 30 16:15:04.405849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:04.405623 4401 generic.go:356] "Generic (PLEG): container finished" podID="f17d6246-6f45-4250-83ab-0002f72feb88" containerID="690067675f69ca31141d2ae68cd6dd2cf2f259ac9da795d3fe612dec3a36dc91" exitCode=60 Oct 30 16:15:04.405849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:04.405772 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" event={"ID":"f17d6246-6f45-4250-83ab-0002f72feb88","Type":"ContainerDied","Data":"690067675f69ca31141d2ae68cd6dd2cf2f259ac9da795d3fe612dec3a36dc91"} Oct 30 16:15:04.405849 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:04.405850 4401 scope.go:122] "RemoveContainer" containerID="020308e90522c04709a49a9759a9be5ec2385597ac60ea1663562aaa83231082" Oct 30 16:15:04.406587 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:04.406300 4401 scope.go:122] "RemoveContainer" containerID="690067675f69ca31141d2ae68cd6dd2cf2f259ac9da795d3fe612dec3a36dc91" Oct 30 16:15:04.406651 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:15:04.406603 4401 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889615-ds4mv_odh-ai-gateway-infra(f17d6246-6f45-4250-83ab-0002f72feb88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" podUID="f17d6246-6f45-4250-83ab-0002f72feb88" Oct 30 16:15:05.457492 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:05.457446 4401 scope.go:122] "RemoveContainer" containerID="690067675f69ca31141d2ae68cd6dd2cf2f259ac9da795d3fe612dec3a36dc91" Oct 30 16:15:05.458133 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:15:05.457771 4401 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889615-ds4mv_odh-ai-gateway-infra(f17d6246-6f45-4250-83ab-0002f72feb88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" podUID="f17d6246-6f45-4250-83ab-0002f72feb88" Oct 30 16:15:09.087626 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:09.087588 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-2rdzt_f4102670-ceac-42d4-8a3c-b685f5d4d43c/manager/0.log" Oct 30 16:15:09.653284 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:09.653247 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-2qndw_7f6e42ec-f48a-4003-abb8-ce3d06af947a/limitador/0.log" Oct 30 16:15:09.820690 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:09.820652 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-dx5wm_1b0b1350-4d9c-4012-8761-ddccf2e06879/manager/0.log" Oct 30 16:15:11.075653 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.075620 4401 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh"] Oct 30 16:15:11.079197 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.079175 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.110957 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.109847 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh"] Oct 30 16:15:11.126586 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.126548 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-sys\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.126868 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.126844 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-lib-modules\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.126974 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.126957 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-podres\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.127083 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.127053 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x88vr\" (UniqueName: \"kubernetes.io/projected/0987334d-7469-4066-9a71-b5dfda5ac978-kube-api-access-x88vr\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.127265 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.127224 4401 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-proc\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.228636 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228601 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-proc\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.228848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228657 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-sys\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.228848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228676 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-lib-modules\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.228848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228744 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-podres\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.228848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228757 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-sys\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.228848 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228809 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-proc\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.229051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228851 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-podres\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.229051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228770 4401 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x88vr\" (UniqueName: \"kubernetes.io/projected/0987334d-7469-4066-9a71-b5dfda5ac978-kube-api-access-x88vr\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.229051 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.228850 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0987334d-7469-4066-9a71-b5dfda5ac978-lib-modules\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.256941 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.256905 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719797fd-5be6-4d5a-9af8-f2c72448cc11/alertmanager/0.log" Oct 30 16:15:11.257237 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.257214 4401 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x88vr\" (UniqueName: \"kubernetes.io/projected/0987334d-7469-4066-9a71-b5dfda5ac978-kube-api-access-x88vr\") pod \"perf-node-gather-daemonset-zjcnh\" (UID: \"0987334d-7469-4066-9a71-b5dfda5ac978\") " pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.390086 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.390057 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719797fd-5be6-4d5a-9af8-f2c72448cc11/config-reloader/0.log" Oct 30 16:15:11.400482 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.400453 4401 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:11.535021 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.534992 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719797fd-5be6-4d5a-9af8-f2c72448cc11/kube-rbac-proxy-web/0.log" Oct 30 16:15:11.691587 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.691514 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719797fd-5be6-4d5a-9af8-f2c72448cc11/kube-rbac-proxy/0.log" Oct 30 16:15:11.718645 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.718617 4401 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh"] Oct 30 16:15:11.726557 ip-10-0-133-222 kubenswrapper[4401]: W1030 16:15:11.720807 4401 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0987334d_7469_4066_9a71_b5dfda5ac978.slice/crio-dc5c941eda75cbab8a7b24a42b9959f7369eda598afc23b7f8afae8f449302ce WatchSource:0}: Error finding container dc5c941eda75cbab8a7b24a42b9959f7369eda598afc23b7f8afae8f449302ce: Status 404 returned error can't find the container with id dc5c941eda75cbab8a7b24a42b9959f7369eda598afc23b7f8afae8f449302ce Oct 30 16:15:11.825772 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.825743 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719797fd-5be6-4d5a-9af8-f2c72448cc11/kube-rbac-proxy-metric/0.log" Oct 30 16:15:11.951448 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:11.951385 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719797fd-5be6-4d5a-9af8-f2c72448cc11/prom-label-proxy/0.log" Oct 30 16:15:12.089008 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.088980 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_719797fd-5be6-4d5a-9af8-f2c72448cc11/init-config-reloader/0.log" Oct 30 16:15:12.409961 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.409936 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h5c85_f324c092-4939-409c-9c05-b95a11343f8a/kube-state-metrics/0.log" Oct 30 16:15:12.525278 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.525242 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" event={"ID":"0987334d-7469-4066-9a71-b5dfda5ac978","Type":"ContainerStarted","Data":"24f01cb545828963d2c80cb3f9203451e8ee63e0bab57b65b4b69d7d2128efea"} Oct 30 16:15:12.525278 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.525297 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" event={"ID":"0987334d-7469-4066-9a71-b5dfda5ac978","Type":"ContainerStarted","Data":"dc5c941eda75cbab8a7b24a42b9959f7369eda598afc23b7f8afae8f449302ce"} Oct 30 16:15:12.525550 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.525449 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:12.536707 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.536683 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h5c85_f324c092-4939-409c-9c05-b95a11343f8a/kube-rbac-proxy-main/0.log" Oct 30 16:15:12.577543 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.577488 4401 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" podStartSLOduration=1.5774755950000001 podStartE2EDuration="1.577475595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:15:12.575085485 +0000 UTC m=+2221.274358653" watchObservedRunningTime="2026-10-30 16:15:12.577475595 +0000 UTC m=+2221.276748764" Oct 30 16:15:12.672851 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:12.672748 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-h5c85_f324c092-4939-409c-9c05-b95a11343f8a/kube-rbac-proxy-self/0.log" Oct 30 16:15:13.067953 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:13.067885 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wkrj_e03e2cfa-48dc-4593-9f45-5192472e0002/node-exporter/0.log" Oct 30 16:15:13.194094 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:13.194063 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wkrj_e03e2cfa-48dc-4593-9f45-5192472e0002/kube-rbac-proxy/0.log" Oct 30 16:15:13.323934 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:13.323869 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2wkrj_e03e2cfa-48dc-4593-9f45-5192472e0002/init-textfile/0.log" Oct 30 16:15:13.449115 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:13.449076 4401 scope.go:122] "RemoveContainer" containerID="351c541d8a2de58467340763f6557cab04e76be8e1504806103fff27205226cb" Oct 30 16:15:14.303607 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:14.303572 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qgtk5_ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d/kube-rbac-proxy-main/0.log" Oct 30 16:15:14.420391 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:14.420364 4401 scope.go:122] "RemoveContainer" containerID="690067675f69ca31141d2ae68cd6dd2cf2f259ac9da795d3fe612dec3a36dc91" Oct 30 16:15:14.439250 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:14.439223 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qgtk5_ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d/kube-rbac-proxy-self/0.log" Oct 30 16:15:14.573410 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:14.573305 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-qgtk5_ccb9f4fc-1e3c-482a-86fc-4a00dff61b5d/openshift-state-metrics/0.log" Oct 30 16:15:15.549339 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:15.549304 4401 generic.go:356] "Generic (PLEG): container finished" podID="f17d6246-6f45-4250-83ab-0002f72feb88" containerID="8c72d96409af45b2c00ba192e8137828bb3c5cec131a7e09071be34a177c71cd" exitCode=60 Oct 30 16:15:15.549339 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:15.549341 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" event={"ID":"f17d6246-6f45-4250-83ab-0002f72feb88","Type":"ContainerDied","Data":"8c72d96409af45b2c00ba192e8137828bb3c5cec131a7e09071be34a177c71cd"} Oct 30 16:15:15.549793 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:15.549396 4401 scope.go:122] "RemoveContainer" containerID="690067675f69ca31141d2ae68cd6dd2cf2f259ac9da795d3fe612dec3a36dc91" Oct 30 16:15:15.549793 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:15.549740 4401 scope.go:122] "RemoveContainer" containerID="8c72d96409af45b2c00ba192e8137828bb3c5cec131a7e09071be34a177c71cd" Oct 30 16:15:15.550048 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:15:15.550020 4401 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889615-ds4mv_odh-ai-gateway-infra(f17d6246-6f45-4250-83ab-0002f72feb88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" podUID="f17d6246-6f45-4250-83ab-0002f72feb88" Oct 30 16:15:15.864103 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:15.864077 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-rzswm_e61536a3-3592-485a-98bd-93cbae66c2cf/prometheus-operator-admission-webhook/0.log" Oct 30 16:15:16.004720 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.004682 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d5b554b4d-f5vcw_007dcfe4-0ea2-43db-b10a-66acd81d4e83/telemeter-client/0.log" Oct 30 16:15:16.164533 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.164463 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d5b554b4d-f5vcw_007dcfe4-0ea2-43db-b10a-66acd81d4e83/reload/0.log" Oct 30 16:15:16.294191 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.294159 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d5b554b4d-f5vcw_007dcfe4-0ea2-43db-b10a-66acd81d4e83/kube-rbac-proxy/0.log" Oct 30 16:15:16.429685 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.429617 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d79665d9-l9kd2_b8b8934a-fc31-4810-9f1e-0e333fc95b02/thanos-query/0.log" Oct 30 16:15:16.558781 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.558750 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d79665d9-l9kd2_b8b8934a-fc31-4810-9f1e-0e333fc95b02/kube-rbac-proxy-web/0.log" Oct 30 16:15:16.609069 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.609040 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv"] Oct 30 16:15:16.699737 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.699675 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d79665d9-l9kd2_b8b8934a-fc31-4810-9f1e-0e333fc95b02/kube-rbac-proxy/0.log" Oct 30 16:15:16.749179 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.749139 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:16.789113 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.789084 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17d6246-6f45-4250-83ab-0002f72feb88-kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/f17d6246-6f45-4250-83ab-0002f72feb88-kube-api-access-xfrtv\") pod \"f17d6246-6f45-4250-83ab-0002f72feb88\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " Oct 30 16:15:16.789299 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.789127 4401 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17d6246-6f45-4250-83ab-0002f72feb88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17d6246-6f45-4250-83ab-0002f72feb88-maas-api-service-ca\") pod \"f17d6246-6f45-4250-83ab-0002f72feb88\" (UID: \"f17d6246-6f45-4250-83ab-0002f72feb88\") " Oct 30 16:15:16.789497 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.789476 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17d6246-6f45-4250-83ab-0002f72feb88-maas-api-service-ca" pod "f17d6246-6f45-4250-83ab-0002f72feb88" (UID: "f17d6246-6f45-4250-83ab-0002f72feb88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:15:16.791598 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.791566 4401 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17d6246-6f45-4250-83ab-0002f72feb88-kube-api-access-xfrtv" pod "f17d6246-6f45-4250-83ab-0002f72feb88" (UID: "f17d6246-6f45-4250-83ab-0002f72feb88"). InnerVolumeSpecName "kube-api-access-xfrtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:15:16.838530 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.838495 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d79665d9-l9kd2_b8b8934a-fc31-4810-9f1e-0e333fc95b02/prom-label-proxy/0.log" Oct 30 16:15:16.891378 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.890673 4401 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrtv\" (UniqueName: \"kubernetes.io/projected/f17d6246-6f45-4250-83ab-0002f72feb88-kube-api-access-xfrtv\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:15:16.891378 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.890718 4401 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17d6246-6f45-4250-83ab-0002f72feb88-maas-api-service-ca\") on node \"ip-10-0-133-222.ec2.internal\" DevicePath \"\"" Oct 30 16:15:16.970346 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:16.970246 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d79665d9-l9kd2_b8b8934a-fc31-4810-9f1e-0e333fc95b02/kube-rbac-proxy-rules/0.log" Oct 30 16:15:17.104228 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:17.104198 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6d79665d9-l9kd2_b8b8934a-fc31-4810-9f1e-0e333fc95b02/kube-rbac-proxy-metrics/0.log" Oct 30 16:15:17.564418 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:17.564386 4401 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" Oct 30 16:15:17.564898 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:17.564450 4401 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv" event={"ID":"f17d6246-6f45-4250-83ab-0002f72feb88","Type":"ContainerDied","Data":"7673a2f8fd0e772f20da8eff95c42f1345644ed90130320c19a7de15bd67ad29"} Oct 30 16:15:17.564898 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:17.564509 4401 scope.go:122] "RemoveContainer" containerID="8c72d96409af45b2c00ba192e8137828bb3c5cec131a7e09071be34a177c71cd" Oct 30 16:15:17.638089 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:17.638057 4401 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv"] Oct 30 16:15:17.651114 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:17.651083 4401 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889615-ds4mv"] Oct 30 16:15:18.543337 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:18.543306 4401 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vkvmm/perf-node-gather-daemonset-zjcnh" Oct 30 16:15:19.432794 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:19.432754 4401 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17d6246-6f45-4250-83ab-0002f72feb88" path="/var/lib/kubelet/pods/f17d6246-6f45-4250-83ab-0002f72feb88/volumes" Oct 30 16:15:21.218580 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:21.218548 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b44548658-8dpd8_0a93c916-efc9-4792-8e71-36faa1839652/console/0.log" Oct 30 16:15:21.390994 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:21.390962 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-krv4w_c37b6ad0-f8fd-4edd-81d9-0cab63ce6368/download-server/0.log" Oct 30 16:15:21.970409 ip-10-0-133-222 kubenswrapper[4401]: E1030 16:15:21.970367 4401 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrn5l" containerName="extractor" Oct 30 16:15:23.932254 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:23.932224 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f29t5m_75b74b6b-5966-4dbb-9e14-92c95de9d211/istio-proxy/0.log" Oct 30 16:15:25.315100 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:25.315067 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-jndpr_afd4958a-0741-47b7-a199-9e51f9cf2e9a/discovery/0.log" Oct 30 16:15:25.421595 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:25.421563 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f6967bb6d-fjrbx_1d1b87c2-0b2c-437b-95db-76f18662d056/kube-auth-proxy/0.log" Oct 30 16:15:25.490925 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:25.488548 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-862j7_672a1ec4-2d4f-4a6f-8e4a-369b6be336c7/istio-proxy/0.log" Oct 30 16:15:25.949725 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:25.949696 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-rsp9f_6cd43595-43d2-4f68-85bb-fea131b69e72/istio-proxy/0.log" Oct 30 16:15:26.091938 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:26.091911 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-szdqp_636add60-0495-48b7-bd72-1cebe229fa0c/istio-proxy/0.log" Oct 30 16:15:26.163844 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:26.163801 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-9dccdc9b7-trg66_8c6639c2-1eac-4716-8a14-cd33938b1c67/istio-proxy/0.log" Oct 30 16:15:26.198859 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:26.198809 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vd8pb_d418480f-1d70-41ce-902f-c36c0c1b4b97/dns/0.log" Oct 30 16:15:26.216697 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:26.216638 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6tt6h_0567028d-7000-45f9-ab89-c4524254517c/payload-pre-processing/0.log" Oct 30 16:15:26.282703 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:26.282680 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-bfqbb_b1a28f78-9ced-4190-9c1f-413f206fd8c5/payload-processing/0.log" Oct 30 16:15:26.334030 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:26.333987 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vd8pb_d418480f-1d70-41ce-902f-c36c0c1b4b97/kube-rbac-proxy/0.log" Oct 30 16:15:26.628037 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:26.628008 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cm4sh_31d9f9bb-0f4f-4110-9ad8-28ac0bbde949/dns-node-resolver/0.log" Oct 30 16:15:28.300936 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:28.300908 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-c99x2_0addea81-c333-4eed-8afa-3762350172bc/node-ca/0.log" Oct 30 16:15:28.734192 ip-10-0-133-222 kubenswrapper[4401]: I1030 16:15:28.734112 4401 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5c79575b6f29t5m_75b74b6b-5966-4dbb-9e14-92c95de9d211/istio-proxy/0.log"